 Welcome to the MetaScience Symposium on Identifying Impact Research Topics. I'm Frank Lieder and my co-presenters are Izzy Gainsberg, Haydn Wilkinson and Cecilia Tilly. Before we dive into the program, I briefly outlined the big picture ideas behind the symposium and explained how the different approaches you are here about can be combined in order to identify impact for research topics. Scientific research is very crucial at this point for the future of humanity because it determines what capabilities we have and it also informs how we will use those capabilities. If we make bad decisions, we could easily destroy the future of humanity. But good decisions on the other hand could create a vast and substantially better future for us. It is clear that, on average, research has been and continues to be highly impactful. It's also clear that the impact varies substantially across different research projects with a small number of projects having an enormous positive social impact but most research projects having very little impact and creating almost no social value. This is partly because there are virtually no principled methods for choosing good scientific questions and finding such questions is really hard. Metascience has so far focused mostly on methods for answering scientific questions and much less work has been done on developing methods for finding good questions. The talks in the symposium are addressing this second problem. We are providing philosophical principles, useful heuristics and rigorous computational methods for identifying impact for research topics. These methods can be combined into a five-step process for finding good scientific questions. The first of these steps is to make a list of potentially impact for research topics by mapping out how science can help us overcome some of the crucial problems that humanity is facing. This approach is called cross mapping. Cross refers to a problem, goal or aim that motivates people to take action. And mapping refers to the process of creating a diagram that shows the various components and how they are connected to each other. So cross mapping is about creating a diagram of the important problems that humanity is facing, potential solution to these problems, obstacles to implementing them and steps that research can take to overcome these obstacles. This diagram shows an example of a cross map. The details don't matter. What matters is that cross mapping starts by listing primary problems such as reducing suffering from mental illness, followed by a second step of listing potential solutions to these problems, such as making psychotherapy more cost effective, followed by identifying obstacles that prevent subsolutions from being widely used successfully. And finally, thinking about concrete research aims that could help overcome those obstacles. For instance, one primary problem many people care about is global poverty. One potential solution is to promote effective charitable giving. One obstacle to that solution is that people don't fully enact their pro-social values and from that we can derive the concrete research aim of developing scalable digital interventions for promoting effective pro-social behavior. This is just a very thin slice of the cross map, but it shows how a primary problem can be connected to concrete research aim through these intermediate steps. When we have done cross mapping, we will have a long list of potential research topics. The winner down, we can then score each of these topics according to its importance, trackability and effectiveness and you will hear more about this process in the second talk. Then we can investigate the most promising topics on this short list in more detail and for that we can use the expected more value of information framework that we will hear about in the third talk. And then we can go from knowing the value of the research to identifying the most efficient way to use limited financial resources and distribute them across the different potential research topics based on cost effectiveness, which is the amount of value that is created per dollar invested. And that gives rise to a prioritized list of important research topics. Now the schedule for this symposium is first we will have a talk by Dr. Izzy Gainsberg on the importance, trackability and effectiveness framework. Then Dr. Hayden Wilkinson will introduce the more value of information framework and then I will give a talk on forecasting the social impact of behavior science on you. And at the end we have a panel discussion that we've motivated by Cecilia T. You are welcome to post questions in the chat. We'll bring some of them into the panel discussion and we may also have time for open questions at the end. And with that I give the floor to Dr. Izzy Gainsberg who's currently a post doctor fellow at Howard County School. All right, great. I'm just going to figure out my screen sharing. Also, we're navigating, I can share my video but I can share my screen but not my video. So you're just going to have to for now listen to me. I'm sorry to interrupt but you should have full sharing permissions now. Are people seeing my presenter view or my normal? We see a slide exactly what we should see. Slides, yeah. And you should be able to do videos as well. Okay, awesome. Well, thanks Falk for kicking us off. I think this is a really cool symposium. I think all the talks that are to follow will be great. I'm excited for our panel discussion too. So, yeah, to begin I just want to, yeah, I just want to highlight that for many of us a core motivation for being in academia and doing science is because we hope to somehow or another make the world a better place through our efforts. So, for instance, in my home field of psychology there's research that suggesting that the most common people that choose, that people choose to study psychology in the first place is because they really believe that it's going to allow them to help others and make the world a better place. And this core motivation, it's reflected in the major institutions and flagship organizations in my fields. So, for instance, the APA, the American Psychological Association their mission statement is to advance the creation, communication and application of psychological knowledge to benefit society and improve people's lives. And so likewise, I'm also in the world of management and at last year's annual Academy of Management annual meeting the theme was creating a better world together. So, although many people want their research to have a positive social impact on the world it doesn't seem like people always choose their research topics in ways that are consistent with achieving that goal. So, I feel like anecdotally people oftentimes choose their research topics on the basis of say what their advisor does what projects offer the easiest path to funding or publication whether a project is consistent with the ideologies they hold or social identities that are near and dear to their hearts whether the topic is trendy. And oftentimes people do choose to study things that they think are important but it seems like it's oftentimes without thinking critically about whether that's going to be the most impactful research that they could be doing. I don't want to altogether discount the merits of those criteria when choosing research topics but if one's goals are to have a positive social impact with their research I believe that there's other ways that thinking about research can help people achieve their social impact goals and so in today's talk I'm going to be discussing this heuristic framework that uses three criteria that can help people prioritize the most important projects and that's importance, neglect and this intractability or for short the ITN framework. And I'll get into the details of these ideas later but before going any further I'll offer some high level definitions of each of these components so importance refers to the scale of the problem at hand which concretely is often something like how many people are affected by the problem and by how much and so sometimes people also refer to this dimension as scale. Tractability refers to how easy it is to solve the problem once thinking about or at least the ease and degree one believes that they can make progress on it so sometimes people also refer to this dimension as solvability and neglectedness refers to how many resources are currently being dedicated to the problem and also the extent to which the problem has the capacity to absorb more resources such as money or time and attention. So where does the ITN framework come from? It was developed by researchers studying the idea of cause prioritization which is a field that's dedicated to figuring out what global issues we should devote our resources to if we want to do the most good with our limited resources and ITN, it's not the only way to compare causes so in many cases it makes more sense to compare potential causes or actions we can take for those causes on the basis of more precise estimates such as cost effectiveness and when I say cost effectiveness I'm really just referring to say the amount of some outcome that people are shooting for per dollar invested in the action also in the world of health for instance it might be the number of lives saved per dollar and sometimes we do have estimates of this kind of cost effectiveness for specific interventions to address specific problems and we can sometimes compare the cost effectiveness when we have these standardized metrics of human welfare and cause prioritization researchers will research these more precise estimates and use them to inform recommendations when possible so for example the graph on the screen here was created by Toby Ord it shows the cost effectiveness in dallas averted per dollar which is a standardized public health metric it shows the cost effectiveness of different public health interventions that were recommended in the 2013 disease control priorities project which is just this massive public health endeavor that tracks the cost effectiveness of different public health interventions and you can kind of see it follows the same curve that Falk was talking about earlier where the most effective interventions are vastly more effective than the typical intervention or the least effective interventions so the kind of data that Ord used isn't always available to help people inform their decisions and that's why cause prioritization researchers developed the ITN framework for situations where the problems being considered don't or can't have precise cost effectiveness estimates attached to them and finally I guess before moving on I'll offer a preview of another point I'll be making today which is that the ITN dimensions they can also apply to what constitutes an impactful research topic or project so just to be clear traditionally ITN has most often been applied to thinking about causes and prioritizing between causes but I'll also be talking about how ITN can help people prioritize between different kinds of research topics so back to the framework again importance refers to the scale of the problem at hand and just to give an example consider the problem of climate change in comparison to ALS which is a neurodegenerative disease climate change it's estimated to cause 150,000 deaths per year currently it's likely to kill millions more in the coming years displacing more than 100 million individuals by year 2050 and in contrast ALS is expected to kill roughly 180,000 people by 2050 so when it comes to scale how many people are affected how severely climate change seems to be a more important issue than ALS and it's not that ALS isn't important it causes patients and people who love and care for them lots of suffering but at the end of the day our resources are limited and working on the problems that are larger in scale can help us have an even greater impact with our careers so how can we apply this idea of importance to our research as individuals so for folks who work on interventions that have direct impact on individuals importance would account for how many people that intervention would reach and the degree to which the intervention would actually help them of course much research aims to have its eventual impact through more indirect means and this research still may be very important in ITN terms for instance it might be important if it informs or has the ability to inform a high importance social issue or cause area such as climate change or global poverty sorry my there we go research can also reasonably be thought of as important if it affects many people in expectation through its ability to inform many different cause areas so such as a methodological or statistical development that can be adopted by many researchers and then applied in many different spaces finally research can be indirectly or directly important if it can be tied to policy and in particular policy around consequential topics and high impact cause areas and that's because policy is often an avenue where many people can be affected so just some examples of what might qualify as important and some of these may seem obvious some of them may seem less obvious so for instance in hindsight research on choice architecture such as whether decisions are framed as opt-in or opt-out maybe this is important research this is a simple idea originally but it was able to suggest a really low cost method for making consequential big consequential behavioral changes in domains ranging from organ donation to people's savings behaviors and just to take another life-saving example some colleagues of mine in the current in-press paper they've pointed to how basic research on the reliability of teams how this was eventually translated to best practices and responses to natural disasters and they ultimately suggest that this kind of basic work ultimately went on to save lives and I guess relevant to our conference I would say that important meta-scientific developments are also important in ITN terms given the role that science will play in addressing many of the world's biggest problems and in so far as meta-scientific developments can improve the quality of science across the board and even make its way into policy and guide best practices meta-science seems to fare pretty well in expectation by the importance criterion and especially if people find ways to work on the most important issues within meta-science so on to tractability which again refers to how easy it is to solve or make progress on the problem one's thinking about I'll again just give another example of two cause areas so consider the issue of farmed animal welfare versus wild animal welfare although there's many more wild animals than farmed animals working on farmed animal welfare may be a more tractable issue because animal agriculture is a human institution that we have much more control over so experts in farmed animal welfare have a variety of policies and practices that they believe would make tangible benefits for farmed animals and on the other hand the world of wild animal welfare it just seems to be way more complex with less tractable solutions from both a moral and practical point of view regarding how we as humans could go about to improve the lives of wild animals so as was the case with importance one way to optimize for tractability in our research is to work on global issues or solutions that cause prioritization experts deemed to be more tractable but we can also consider specific dimensions that may be relevant to tractability so for instance this could include whether the research topic is super complex or difficult which might make something less tractable even if it might be more appealing to a scientist who like to ponder over complex problems how much time or money would be needed to really make progress on the issue whether the topic would have a difficult time being accepted by the public even if we were to able to make progress on it from a theoretical standpoint this might still be a barrier and make a solution or a problem less tractable overall and also a problem might be less tractable if one as an individual who wants to work on it doesn't really have the expertise that's needed to make progress on the issue so these ideas can be integrated into models to rate tractability so I'm just going to give an example of one scholar who I saw did this Paul Stern who's an environmental psychologist developed a model rating the effectiveness of potential environmental behavior interventions and his model included a term that to me captured tractability even though he didn't call it that he broke this term down into three components that were kind of multiplied by one another one was technological potential and this referred to how much the relevant technology that was a part of the environmental behavior intervention how much the technology would actually affect some climate or environmental outcome so for instance we might think about buying an electric car versus recycling these two kinds of behaviors have very different environmental outcomes then we have behavioral plasticity which was how much we could reasonably expect people's behavior to change so even if we can make change even if we can make a given change we might not be able to actually change people's behavior the behavior we want to change and then initiative feasibility which is kind of like a macro level consideration of behavioral plasticity which is kind of the systems level assessment of whether some intervention could reasonably be implemented so I'll finally move on to neglect in this which as I mentioned earlier is how many resources are currently being dedicated to a problem so more neglected problems can typically absorb more resources and therefore they might have more low hanging fruit for high impact research this is connected to the idea of diminishing marginal returns if there's already a lot of people working on a given problem it might be over saturated and there might be diminishing marginal returns to work on to work on that topic so just to take an example of neglecting this again from like comparing two causes I guess we can look at data from 2010 which shows that mental health had a global disease burden at the time that was over twice that of HIV but it received under 2% of the development assistance development assistance globally as HIV suggesting that again at least relative to HIV mental health was being somewhat neglected and if we want to figure out how to assess neglectedness some possible routes are examining what kinds of topics are or are not being funded and also maybe analyzing what topics people are or are not publishing on or presenting on at conferences and so as is the case with importance and tractability we can also analyze neglectedness when comparing broader cause areas and we can also think about neglectedness within a given field so just to take an example of this in a recent paper that some colleagues and I wrote we examined how many publications per year there were on various mental health topics and compared that against their global burden of disease to get an idea of which areas within mental health were being neglected relative to their overall scale so moving beyond ITN for a moment I just want to highlight one more very important ingredient to the recipe on how to do the most good with one's research and that's the idea of personal fit so the ITN framework it was originally used to kind of it was originally thought of from maybe a societal standpoint and how society should prioritize different cause areas and we can also think of it from an individual level perspective but when we think about the individual perspective it's also really critical to think about how we as individuals fit into the overall equation not just these dimensions that we think about in terms of the research topic or the cause area because as soon as we consider ourselves it becomes obvious that the most effective behaviors aren't the same for everyone and they depend on the person so as a trained social psychologist I'm not really well suited to work on the technical elements of carbon capture and that's an obvious example but the broader point is that we can all have a bigger impact when we're able to leverage our unique skills, knowledge and expertise when we're able to leverage those things in our research and that's because when we have those things it gives us the tools to unlock the most powerful behaviors and because that provides us with the capacity to take actions that others would be unlikely or unable to do it gives us kind of a higher replacement value also from a personal fit standpoint we're also going to have a greater impact when we choose topics to work on where we can sustain our motivation it just doesn't make a ton of sense for someone to work on even a really important topic if they're not going to be able to motivate themselves to do quality work on that topic so personal fit and their own kinds of research decisions is always an important part of the equation so before moving on to the next talk I'll just post this slide here so this lists cause areas and global issues that cause prioritization researchers have identified as being potentially high impact areas to focus one's work and so for the most part these are recommended because they fare well by the importance tractability neglect and the standards and to be clear lists like this they're always being updated as people do more research and the world changes but you know this is kind of a working list of some of the cause areas and topics that fare well by these ITN standards so that's all I have for now I'm just going to say a quick thank you to co-authors who contributed to some of the ideas I presented today to Falk for organizing the symposium and our other presenters and now I'll hand it off to Hayden who's a current postdoc in philosophy at the Global Priorities Institute at Oxford University Thanks Izzy, thanks to the organizers and thanks everyone for joining Falk and Cecilia can you just give me a wave if you can see my slides and it all looks okay okay so those of you watching there's a good chance that you're a researcher yourself who wants to make a positive difference in the world through your book indeed to make the greatest positive difference you can so you face a practical ethical question if you want to have as great an impact as possible what research topics should you pursue and more generally when we're deciding what research to do how can we assess the impact of different research topics at least the impact that's actually relevant to the decision making research topics so as to maximize my positive impact what's the criteria for me choosing the best one and can we say anything about which topics those actually are at which topics are most impactful can we narrow it down and perhaps use any simple heuristics to choose the most impactful topics so these are the questions I want to try and answer in this very short talk I'll briefly run through my current tentative answers but first a quick note on what I'm not doing here I'm not arguing that we are required to do whatever research is most impactful that this is some sort of moral obligation I'm not arguing that anyone is doing anything morally wrong by not doing some research and I'm not talking about this separate issue of what something bodies should do whether if anybody should use the same heuristics that I previously or the heuristics that he was just talking about I'm just talking about how individual researchers can figure out what projects they should do if they're in fact motivated to do good through their work so the question that's actually relevant for most people listening to this now to give an answer to that question first what is the relevance of impact well some researchers might be focused on particular sorts of impact but for the questions I'm interested in I'm concerned with the broad moral impact how much better the world is as a result of the research being done I take that that's what say impact assessment and so on I'm trying to capture but they usually don't do a very good job of it I'm also not as interested in how much better the world is guaranteed to be after all we're always going to be uncertain of the results and the impact of the research and we don't want to say trivially but no impact no research has any impact it's never certain so interested in how much better the world is as a sort of gamble given your uncertainty given since that's the situation we're in then we're deciding what to do to give an answer to the question of what research is most impact well whatever research results in the greatest expected moral value given your uncertainty and by value here I mean moral goodness in a very general sense not specifically money or well-being just how good the outcome is according to whatever theory of moral goodness is for this isn't a very satisfying answer it's no better than just saying what research is most impactful whatever research the correct moral theory says is most impactful I think we'd like a stronger practical conclusion here we'd like some idea of which research this actually is we'd like some helpful theoristics to know after all that's what coming bodies try to do they're not trying to do a fully general moral analysis of each project and others so we'll run through a few possible theoristics some of which gets used by coming bodies so here's one, the size of the audience reach more people the work reaches the greater its impact and it often is better if research reaches more people at least sometimes that means that it has more opportunities to make a difference in the world but I want to say a large audience by itself doesn't guarantee that the work actually makes the world better so here's an example of it not doing so at least not doing it so I suppose a historian researches the life and rule of the 9th century English king offered the great they learn various salacious details about Alfred's personal life that of his courtiers and various lords then once this work in history is published a novelist reads it and recreates many of those details in a novelization their novels are then adapted perhaps into a television show which is viewed by millions of people those people are briefly entertained but otherwise go about their lives as normal and if it weren't for that book and the historian's work then a different show would have just been made instead that was equally entertaining so there's actually a real world example I haven't read the book or watched the series so I'm not sure how true to the historical details it is but you can imagine the story roughly going like this and it doesn't seem like that's made the world much better merely because many people have watched the show or read the book and yes I've read through a fair few impact assessments myself quite a lot of people when claiming that they impact his research but that research is impactful say something like this well it was picked up by mainstream media outlets or it was novelization and then many people were attracted to it it doesn't mean it made the world any better so another case I'm not heuristic is that if some phenomenon is impacted there's a big problem in the world and any work about it must be impacted this makes the mistake of instead of measuring the impact of research you're doing you just look at the impact of the phenomenon that you're researching something so here's an example so you study as a biologist you study the relationship between climate change and the breeding patterns of the eternal roly poly bird you find that rising temperatures decrease the roly poly bird's birth rates and as a result one and a half degrees of global warming as is anticipated will drive them extinct even with conservation efforts and let's suppose that governments, voters, corporations, consumers don't especially care about the roly poly bird at least the survival of this species despite being a very bad the extinction of the species being a very bad thing the survival or extinction of it will not tip the scales for any of those decision makers to do things differently and reduce their emissions so the same amount of emissions will happen the same harms happen the bird still goes extinct regardless of whether you make this discovery so again this doesn't seem like you've actually had an impact even though you are researching something very important or very kind of a big problem but at the very least if your research is a guarantee not to change that outcome then it's not the highest impact research you could go so there were two heuristics which I think get closely used a bit and I've got a heuristic that I think does better than either of them but before I describe it I'm going to give you it lands up nicely with what genuinely makes the world better so suppose you live in a world where many people eat meat including pork and they do so largely out of ignorance that pigs are highly intelligent because perhaps science just hasn't figured this out and as a result 800 million pigs are kept in conditions of suffering at any given time and this is all true to life except we have already had the research shown that pigs are pretty hot so in this hypothetical where we don't yet know that you study the intelligence of pigs they arrive at small human children in their same problem-solving ability, memory and so on and your findings are then picked up by an animal advocacy organization communicated to tens of millions of people and many of those millions of people reduce their meat consumption as a result and millions of your pigs have horrible lives on the factory farms so just assuming that pigs suffering buzz matter morally which you might disagree it's pretty plausible but this argument at least you think that is if you think pigs suffering is terrible then in this case you've got an enormous positive impact by doing this particular research because that just so happens to be the bit of information that many people's decision hinges on and those decisions have high stakes there's another case here which I'm going to skip over but it works basically the same this is more of a realistic view of this so heuristic but the important picture of the pigs' case which wasn't present in the previous two cases how can the research actually be impactful is that the research provided useful decision-relevant information to real-world agents and that that information potentially meant to change their decisions to greater effect so in effect it was a case where the research in question has high value of information and that's the heuristic problem so the value of information is a concept from decision theory and economics of how much it would be worth for a decision maker at least a rational one to have the findings of that research prior to making their decision and we can make some quick assumptions about what constitutes rationality here standard theory of this expected value theory which says that conditions of uncertainty best action or the best option for you is whatever brings the greatest expected amount of value so the expected amount just means the kind of average amount where you weighted the average based on the probability of different types that'll maybe become a bit clearer in a second so if you accept that standard theory you think the expected value of things is what matters then you can evaluate the value of a piece of information or of a piece of research so the expected value of doing that research and the probability that research will change your decision multiplied by how much improves your decision or more completely the value of information of some piece of research is this summation we're summing up so for each possible thing you might find each F the probability of that F multiplied by just how good it would be to discover that thing how much it would improve your decision and you're summing that over all possible findings that ends up taking the average benefit of having done the research okay so here's an example and here we're just looking at the value of information for a single meat eater who doesn't think he's very intelligent and includes as a result of that that person how valuable is it to have this research about pig intelligence well here's our equation you can plug in the value of information of this research equals the probability that pigs are indeed smart and let's assume that if they are smart you'll find that out multiplied by how good it is to not eat them if they're smart or how much better it is to not eat them and presumably that's quite a lot they're very clever creatures just like the badness of eating a small human child of similar intelligence that's much better not to eat them that's a lot of value that term is going to be quite big plus the probability that pigs aren't smart whatever probability that is multiplied by zero because you're already eating meat so if you find out that pigs aren't smart and you continue eating meat that's just as good so as long as this probability at the start that pigs are smart is pretty high or you're pretty uncertain of how the research will turn out conditional on them actually being smart it's a lot of value and this total value of information ends up being quite a large amount it kind of looks very good in expectation terms to do the research okay the value of information at least as it's usually used and as I find it is in problems so it doesn't really work in the general context of doing impact research here's some problems with it so first in doing research you're often providing information about one decision maker it's not just a simple decision by one person these different decision makers will often face different decisions with different stakes so again like that's going to further complicate it you can't just say okay this is a pretty decision maker so you multiply the value of information that you make nope they all have different stakes and they're different positions also even if something is true your research might not succeed in showing it some effect sizes are very small or you need like an extremely high power to detect the relationship or whatever also at least as I put it my other information didn't capture the potential harms of finding evidence for false conclusions after all you might find just by chance in your trial you find evidence for something that's impact false you find some slight positive evidence that drug A is really it's more effective than drug B in fact it was just a chance of results the one in a hundred results that are just by chance from the sample the control group the treatment group ended up better off so VOI didn't capture also expected value theory might be the wrong theory of rationality but that's something I'm not really going to talk about here and what I'm about to propose can be just kind of modified to fit with other theories of rationality but expected value theory is kind of the simplest one so stick with that so my proposal is to still use something like value of information to modify it or this particular study so here's the proposal so expected moral value of information for all or can be for which sounds kind of like for all of you this is how much better given your uncertainty the outcomes of all those decision makers choices will be if you do the research about how many such decision makers there are what the spec spec decisions are how many of them will listen to you perhaps that's sometimes not too many and you're uncertain about what the findings will be so I'm going to make some assumptions again expected value theory and if we have the right sort of theory of moral value this will, this equation will vary if you have a different theory of moral value but it keeps it simple so again beef of some bit of research would then be the expectation so on average how many decision makers will there be who will be affected expectation of the stakes of their decisions relative to and then we've got the same sort of terms before so you're taking the expectation you're doing this probability weighted sum now you're summing over each possible finding f and each possible truth t sometimes what you find is not what's true it might be different things so then we take the probability that you find f if t is true multiplied by how much better it is to find f if t is true basically the value of information taking into account the risk of risk of what you find not being what's true and then like weighting that up by number of decision makers so the question are the most impactful topics in any given field the ones with highest mp and that's kind of a tough question but to try and answer it we can ask the people about what are all the different ways research makes the world better and is the biggest source of such benefit by providing evidence which will have more evidence more useful evidence so one way is that perhaps knowledge is intrinsically valuable the world is better people know more things or if we as scientists know more things even if no one is happier well I think no matter what research you do you're going to generate knowledge so just knowledge is valuable in itself and that doesn't really branch high also personally my intuition is that if you're comparing the value of knowing more things versus fewer people living in poverty or people having better lives and experiencing less suffering less suffering and so on that is far more than the bit of extra knowledge so I don't think this is going to be where most are divided so the source of value here is that research being done is in itself intrinsically valuable you might think if the world is full of Galileo's tinkering away in their workshops and so on that just is like a morally better world than a world of people watching Netflix shows I think this has the same problem whatever research you do you're going to be doing research you're going to be producing the same intrinsic value by research being done likewise I don't think this compares to the value of connecting people from experiencing great suffering so I don't think this is going to drive too many choices with what we do with research so this is the one I think matters most well research provides evidence to agents to make morally better choices and this includes providing the knowledge necessary for new technologies which effectively reveals new better options solving problems and the research that does well in this respect but has a lot of impact in this way is just research that has higher input so the more input the more stuff I think there is a fourth root to impact that maybe rivals and be so research can also manipulate or motivate or persuade or inspire people to make morally better choices without providing new evidence and it's unclear to me that we do more good by providing useful evidence to people and by just participating through our research so you might think the science is clear the sufficient evidence that smoking tobacco really really bad for you and it's not worth it for any agent to do so and yet you might think that continuing to do a bit of research on the harms of tobacco smoke is useful because it gets the occasional headline it reminds people on the street that smoking is bad and maybe that reduces smoking rates unclear but maybe that's a bigger part of the impact there are some reasons not to think that this is a great way to go one is that it seems a little paternalistic to do research just to persuade people where researchers, we're not marketing professionals and there might be something morally buggy about throwing in with that poll these effects are also riskier so when it comes to providing evidence we get more evidence when we get closer to the truth whereas we don't necessarily get more persuasive when we get closer to the truth so we might inadvertently do a lot more harm that way also I think as researchers we often don't know what will be most persuasive to every person so I think we might still be often topics will be tied in their persuasion value whereas they're in deeper marketing for them obviously also as researchers we just want to discover new things we don't just want to display if we want to have impact and specifically we want to have impact through discovering new things and informing people and maximising in people there's still a pretty good risk but there's some uncertainty there in some cases you might be motivated and inspiring okay very little time left but I'll just run through one very quick objection which is that if we're going to try and maximise impact in general and we're picking our research that seems to rule out blue skies research or research that's not driven by curiosity and what's interesting which might end up having more impactful results in the long term most of the great results were largely driven by scientists who were serious this one paper that impacts driven research seems to rule out the possibility of open-mindedness in the very start statement by the British Philosophical Association say that impact is long-term unpredictable hard to quantify and not the basis for measuring quality of research and I think it's pretty close to right when it comes to impact as measured by agencies but I think MPFA is a bit too complicated considering the perspective of individuals so I think in practice maximising in people will often involve pursuing pretty abstract work with only low probability long-term impacts because after all it's probability multiplied by value you can have a very low probability value is enormous and that's still high on people I think many of the best options will do that also probabilities involved is of successfully finding something out which do depend largely on the researchers curiosity and interest probably what will be recommended are topics that the researcher themselves are very interested in and then just among those once you pitch the interesting topics you narrow it down to what is impactful among those interesting topics because it's the only topics you may want to progress on anyway there's a further supplement to that so what are the takeaways if you want to do the most impactful research you can what should you work on the answer is not clear but if you want to choose an impactful research that's not the most impactful but you just want a good jurist to be sure that you're doing something to be impactful a topic with high MPFA is a pretty good choice but there's this table how and as do you figure out the probability of all these things how do you quantify all these numbers well with difficulty and I can't believe that Falk was about to answer that in his talk so I'll head over to Falk Falk would you like a quick intro well I'll give you one anyway Falk is a computational cognitive scientist at the Max Plan Institute for Intelligent Systems where he runs the rationality enhancement group and he'll be talking about core customers in fact your favourite science currently over you Falk Falk you are muted still hey can you give me an arm yes thank you in the talks you heard so far you learned about the concepts of course mapping the ITN framework and the moral value of information framework and these correspond to these first three steps of this overall method for identifying impact for research topics I will now provide some practical computational tools to apply the expected value moral value of information framework to real real decision so we will research topics which involves some very difficult forecasting problems of identifying what the findings could be and what the consequences of those findings would be if they were rediscovered in addition I will illustrate the application of this method to a first proof of concept research topic and share the preliminary results of that analysis with you and then I will lay out how funding agencies could use social versions of this method in order to set research priorities so how can we forecast the moral value of research as Hayden pointed out such predictions are difficult but not impossible two concrete difficulties are that the pathway to impact can be very long indirect and highly uncertain and the second difficulty is that some research doesn't only generate information that informs our decision between options we already know but may also create entirely new options that we didn't even consider before to overcome these problems our computational methods proceed by first building a data driven probabilistic causal model of the research projects pathway to impact and then we use historical data to predict how likely more research on the topic is to achieve different amounts of progress and we implement this using probabilistic graphical models and molecular simulations using the modeling tool and guesstimate to apply the moral value of information framework we assume that moral value consists in directly or indirectly reducing suffering or increasing well-being moreover we assume that everyone's well-being counts equally and under these two assumptions we can measure the moral value of research by the number of well-being adjusted life years that it saved or that are created as a consequence of the research although this approach is very general the pathway to impact is very different in different fields in the discipline that I'm most interested in personally is psychology therefore the initial scope of this project is behavior science research in particular behavior science research that either evaluates an existing behavioral educational psychological intervention or might improve one or might lead to the development of a new potentially more effective intervention for these kinds of projects the pathway to impact can be modeled roughly as shown in this diagram where the first step is that the research produces, improves or evaluates one or more interventions for changing people's decisions, behaviors character experiences beliefs or attitudes and by improving our knowledge for repertoire of these interventions we are then able to have more positive influence on people's behavior and consequently we can increase people's well-being either directly for instance by reducing their suffering from mental health issues or indirectly for instance by changing people's acceptance of certain policy proposals then lead to policies that change people's behaviors in beneficial ways that benefit future generations and so on because the interventions produced by behavior science generally require some amount of money to deploy we can think of the increase in well-being as the product of how much more cost effective or best methods for doing good become as a consequence of the research and how much money will be invested in doing good and these kinds of interventions in the future if I study for this modeling approach I will apply it to Rachel Baumstegger's research on promoting for social behavior in this case the research pathway to impact can be modeled as racial conducting research and for social behavior results in a new online intervention for promoting for social behavior in emerging adults the deployment of that intervention then increases the frequency of for social behavior and whenever someone engages in for social behavior it increases their own well-being and it also generates some happiness in the person whom they are helping and together these different effects which persist over time as for social behavior is not only increases immediately but also over the following weeks and months then increases the total well-being of humanity we implemented this model the practical model initially in guesstimate this is a screenshot of this model each of these squares is a random variable each arrow corresponds to a causal effect between these random variables one component of this model describes the effectiveness of the intervention increasing kindness the frequency of kindness and the effect of kindness on well-being another component describes the cost of knowing the intervention and a third component combines these two pieces to estimate the cost of kindness with which the new intervention can be used to do good zooming in on the first part it has one component that describes how the intervention will increase the frequency of for social behavior another component on how for social behavior increases well-being and a third component that combines the frequency of the intervention the benefit into the total benefit over time now let's zoom in into one of these nodes in this probabilistic graphical model get a better sense of what that means so here we see that this particular variable the initial effect of kindness on the person performing the kind behavior is modeled as a normal distribution with a mean and standard deviation derived from meta-analysis of previous studies on the effect of social behavior on the well-being of the person who engages in it now having such a probabilistic model we can ask a number of interesting questions such as how much more value did this project create and how much more value could we create by evaluating this new intervention in a large randomized controlled trial and also how much more value might a new project that is just being proposed be able to create if the funding agency decides to support it so now let me illustrate tentative answers that our method gives to these questions for the case study of Rachel Baumstegers project our method suggested that there is still a large amount of uncertainty about how cost-effectively this intervention can do good in particular there is the chance that this intervention could be extremely effective and a high probability that it will be only moderately effective compared to other existing interventions the new intervention shown in purple here has most of its probability below the non-effectiveness of the best charities for promoting global well-being but it also has a very long tail meaning that there is a chance that it might be substantially more effective than any of them so that suggests that it could be very worthwhile to evaluate this intervention in a randomized controlled trial but a randomized controlled trial is also very expensive so it's a small chance of this intervention being effective really worth it to invest in an expensive randomized controlled trial our method says yes it would be very worthwhile to do so in fact if we randomized controlled trial for costing up to $300,000 that would generate $330,000 well-being adjusted life years in terms of moral value so be extremely beneficial to do so despite the high amount of uncertainty about what exactly the outcome of the randomized controlled trial would be it is quite clear that this would be worthwhile doing in fact this randomized controlled trial would generate almost one well-being adjusted life here for every dollar that it costs and that's more than 10 times as effective as the best charities in the space of global health and well-being so assuming that is a intervention that Rachel Boncher developed will be evaluated in a randomized controlled trial was it worthwhile for her to develop it and if so how valuable here our method says the research and development project actually conducted was in expectation worth 360,000 well-being adjusted life years that's again an enormous number and it is so high in expectation because of the possibility that it might be more effective than the best existing intervention because if it were it would invest billions of dollars worth of philanthropic resources in a more effective way to do good and that could save so many more people by contrast if it was less effective than the existing interventions then all we would have lost is the amount of money that was spent on this one project to develop and evaluate intervention afterwards we could deploy fully to the interventions in order to answer this question we applied our method with a model that only uses information that was available before Rachel started her project so we predict how effective our intervention could be given historical data on previous interventions and we conducted a basic meta-analysis to estimate how long the effects of prosocial cave interventions typically last and we found that our method did indeed predict that this project would be extremely valuable in fact our analysis predicted an even higher amount of moral value from this project than our exposed assessment suggested but most importantly both the a priori prediction and the a posteriori analysis concur that this project was extremely valuable now assuming that future versions of our method that I just illustrated with one case study will be able to reliably make accurate predictions about the social returns and investment for different research projects then funding agencies could use it to set research priorities an important step in deciding how to allocate limited financial resources across many different projects to do the greatest amount of good in total would be to rank all of these potential projects by the cost effectiveness and first fund the projects that are most cost effective at doing good and then move down the list this requires being able to quantify the cost effectiveness of research projects our method can easily be extended to do that by dividing the MV file of a research project by the expected research costs and once we have done that we can then rank all of the projects by this cost effectiveness ratio concretely yeah, we can calculate the cost effectiveness of research in topic x by the MV file of research on topic x divided by the expected cost of conducting the research that gives us the project's cost effectiveness in terms of well these per dollar now um how cost effective was Rachel Baumstegers project compared to donating to a highly effective charity the answer is it was extremely cost effective it generated about 340 well being adjusted like this per hour of research time and that corresponds to about 9.2 well these per dollar and that is more than 10 times cost effective as the best charities in the global health and well being space could we have predicted that to answer that question we again um predicted the more value created by her project using only information that was available for she started working on the project and we estimated the cost of the project based on how much money was um given to similar projects by the template charity foundation in the past we found out the prediction for the project's cost effectiveness was remarkably close to the um cost effectiveness of the project this is very encouraging for the methods potential ability to predict how cost effective research projects are going to be so in summary in these four talks you've learned about four key concepts for identifying impact of research topics including the ITN framework the more value of information and forecasting the cost effectiveness of projects in conclusion we can quantify the positive social impact of intervention research and like the other types of research as well the additional work um it is even possible to accurately predict in some cases how cost effective research on the topic is going to be and future work will extend this approach to identify the impact for research topics in terms of their cost effectiveness at generating social value and we also observed that research on promoting for social behavior in particular could be highly impactful I hope that in the long run this line of work will shift the priorities of behavioral science research and criteria that funding agencies use to decide which projects to fund towards research that is highly beneficial for the long term flourishing of humanity if you'd like to learn more I invite you to check out these resources and if you have any questions or suggestions or like to get involved in this project you can reach out to me. Thank you very much Thank you so much Falk and we will now yeah my name is Cecilia Tilly and I will be moderating the panel discussion that we will have now and I'll also mention we have this Q&A function we have gotten a couple of questions there but if you have more questions you can enter them there and I will keep an eye on it and include your questions as well in the panel I think something that's interesting here is that there's clearly like less action in the meta science community around this question about how to choose what we do research on compared to things like transparency and research quality and there is some tension there potentially with academic freedom and like who should actually be setting their research agenda and Haydn I'd like to start with asking you actually because you you started out with saying that your method is not used for funders or it's not meant to be used by funders it's meant for individual researchers who already know that they want to choose their topic based on what's most impactful but I wonder do you think this is the right or most effective like level to address this problem it might be hard for individual researchers to go about this freely unless they have backing of funders for example or their institutions yeah I mean in the ideal world it'd be great if just if the top down approach was possible and you know all researchers in the world could work on the most impactful projects available to them and the research funds could be allocated in a similar way or perhaps not but at least on the face of it it seems like good at an institutional level to have more of a shift toward the impactful research being funded but I just personally can't see any way that we could achieve that so at least with the the MVPA so either with the MVPA framework or using importance trackability and neglectiveness there's a lot of subjective assessment to be made there and I imagine it would be very easy it'd be very easy for people who just want to research a particular topic because that's the topic they're personally interested in then on a post hoc basis rationalizing that to a funder and making the case that it's extremely impactful when it's actually not so impactful according to their own subjective assessments because an external funder unless they themselves unless the grant maker is themselves an expert on the topic or as expert on the topic as the person proposing it they're not going to be in a position to know how trackable it is where this little bit of research might lead they're also not going to be able to assess all the relevant probabilities this researcher will make this breakthrough it's also perhaps relatively easy to just invent scenarios where by some huge stroke of luck a particular research project goes on to have an enormous impact even though in practice that's not actually likely to happen I just I can't see a way where I would fast research funding bodies to do this well or in a way that isn't easily then gained by individual grant applicants Interesting, we have a related question here that I've gotten a lot of upvotes also from some attendee who says that their experience in everyday life as a PhD and postdoctoral researcher is that it's difficult to work on topics supervisors do not work on and reasons include their expertise their network, different beliefs about what impact is and so on and that it's hard to get funding for topics that's not currently considered important by society and funders so they are asking if any of you who are speakers have any tips on how to address these challenges on a individual level and how to get to actually do the work that you think is most impactful I think you can take a fast cited approach and work towards building the skills and developing the ideas for the time when you will have enough autonomy to embark on the project that you have identified to be most impactful if you're already committed to a particular PhD project and you don't have the way within that then you can at least prepare yourself for the next steps after your PhD So rather take your time Mr. Yes, life is long and what matters is the impact you have over your lifetime Issey did you have something to add here? Yeah, I mean I think that's a very common problem and I think the advice might depend on the field that one's in and the latitude that they have given their field or their situation but at risk of helping people advance moral values that I don't agree with I would suggest that people read about useful negotiation skills and essentially trying to figure out, okay how can I frame my important topic that I truly believe is really important in terms and values that are aligned with those of my advisor or my institution or the funding agency and there are oftentimes more points of overlap than people may realize on the surface but there can be space to make it work in those situations if people really think about how they can negotiate in very broad language there Thank you Falk, I think you had a little bit of a different angle in your presentation though like Haydn, you said you were pessimistic basically about funders being able to make these assessments well but Falk, your presentation seems to be more addressing but actually funders or grant makers could use these tools Do you have anything to say about why you are optimistic that you think this could work? There are some funders who already employ researchers to identify important research topics such as opens and other people I think such funders might be open to professionalizing their research methods for setting research priorities if we can make our method sufficiently easy to apply and train a sufficient number of people then at some point funders might be willing to employ people who have been trained in those methods or might be willing to facilitate workshops to their own course prioritization researchers I think this is a low probability event but it would be very impactful and I think it's worth working towards it Another question that I think is more on the funder level or maybe also on the individual level that is about when we do prioritization many of your examples were choosing between different projects or different research questions within a specific field What are your thoughts on comparing different fields when we're talking about funding allocation it might be relevant to think about how much should we allocate to behavioral science in comparison to physics research for example What are your thoughts on trying to do this in a systematic way and do you think the methods that you're proposing would apply there? I might just give a quick easy answer to this question which is that as Falk's calculations demonstrated the cost effectiveness of a lot of research is gargantuan compared to a lot of other ways to spend money especially if governments are deciding or maybe we want to save some money this year we'll cut down the R&D budget or the funding for universities I think there's a wrong case to be made there that just in general rising the tide of research funding is a pretty good bet as far as which specific fields to give the money to I mean I as a philosopher am not qualified to figure that out but maybe we want to be able to take on this I agree I think our method can be used to make a strong case for more research and R&D funding and that's probably more important than researchers arguing with each other about how to allocate two little resources between important projects that said I think the same approach that I outlined can also be used at the level of research fields and from there on also at higher levels of disciplines even where the discipline or the field as a whole can be as impactful as the most impactful topic within that field and everything can be compared in the same currency of how much increase and where it will be as a consequence of the research and whatever field it is per dollar of research spending so it is possible Does any of you think that anything other than well-being should be taken into account here like I could see a lot of researchers thinking here that there are other things as well and that well-being if we compare to charities for example that research has other objectives as well and there are things like understanding the universe or understanding the origin of our species for example that might not be easy to translate to just increased well-being but you are all pushing pretty strongly for the well-being angle. Do you have any doubts or do you think this is the sole purpose of research? I would say not just well-being I mean there's plenty of other values that seem potentially morally important so if your research can persuade decision makers or can provide sufficient evidence to decision makers to correct some great injustice that seems like possibly also very high priority and as far as even within well-being there's a lot of different kind of sub-questions you might think that preventing suffering is far more important than increasing happiness of people who are already pretty happy so deciding whether to assessing the value of research that might just improve quality of life in somewhere like Denmark even more versus research that might be especially research that might potentially benefit pigs and chickens on factory farms so if you are clearly living far worse lives than especially rich well-looked after humans then there's also issues there of what you prioritize also when it comes to yeah so there's one potential value of the well-being of currently existing people or the well-being of people who will inevitably exist in the future there's also the potential value of bringing more people into existence or saving humanity from its own disruption or something or likewise the dis-value or amenity not existing and there being no well-being in the first place so these are kind of different values which you might value either of them to a different extent so well-being isn't cut and dry one concept I think conceptually most things that people value and care about can be derived as valuable in terms of the well-being that results from accomplishing those other values so at least at a fundamental philosophical level we could probably derive all of the other values from well-being so in that sense this thing could be inclusive but I also want to be clear that I don't think that it ever will be the case or should be the case that all decisions will be made exclusively based on well-being but because we are so far away from it I'm not worried that this will happen anytime soon so at this point I think we have more to gain from making more decisions based on well-being considerations than fewer and my concerns will shift when 99 or 95% of decisions are made based on well-being I can't say I have anything to add above and beyond what Falk and Hayden said they kind of echoed maybe the things that I was thinking Thank you I can follow up anyway with a question that I think was asked during your part of the presentation that was from Larry in the audience who asked if there has ever been a hind-cast analysis of cost prioritization for previous research programs for example penicillin polio vaccines or the green revolution Yeah there's definitely been research on people's estimates of the cost-effectiveness of that research or the estimates of the amount of lives that were saved as a result of doing that research and I don't have numbers off the top of my head I don't recall them being extraordinarily large and kind of like in line with the value Hayen and Vifa obviously so people do look to the past and kind of try to think about I don't know how much of that research is specifically looked at the costs that were involved I know that they more typically think about the costs that were saved but the number of lives that were saved were so high by some of those high value health interventions that they I would presume that they would dominate the costs that were involved I don't think I don't know and I'm not a cause prioritization researcher myself my role and my thinking was when I started becoming interested in this to help people choose research topics in psychology and the behavioral sciences so I can't tell you what else has been done I don't know I don't know how people were thinking about the concept of cause prioritization back then so the angle of the question is how were people making decisions on what to research at the time that these interventions were being developed I'm not sure I don't know the answer to that and I don't know if people have taken a retrospective analysis or historical analysis of how people were actually going about cause prioritization 100 years ago or 80 years ago so that's a good question and I'm not sure I hate it and you might know there's some work that might be interested in that assessed retrospectively the social returns on investment for Able Science, Social Science, IND funded by USAID's Development Ventures Innovation Fund there's a paper by Richard Kremer on that and there's also work on the cost effectiveness of past agricultural research in the developing world these are the two examples of historical analysis of cost effectiveness of past research that I'm aware of thank you we have another question here from the audience that is touching on that partly that society today may not always appreciate or realize the topic or the potential impact of your research and they're pushing for that basic research maybe doesn't have a clear applied value yet but that might still be very valuable in the long-time scales and I think this relates a bit to something you said Haydn that you actually would expect using to lead to more of this kind of low probability high potential impacts kind of research could you explain a bit more because that sounds like you think this would actually promote basic research yeah although I admit that's pretty speculative so when it comes to when it comes to striking a balance between research that has like very direct very high probability predictable impacts and research that who knows how it'll turn out very basic research research is very kind of separated from potential impact we definitely want to strike some balance between the two we don't want all research to be of the former type we don't want to say only the former type is impactful and likewise we don't want only research of the latter type or we don't want like exclusively basic research happening including research where there's no conceivable way that it would actually pay off yeah of course it's going to be very hard to figure out of the basic research topics which are the ones that have this small probability of an enormous payoff and which ones just have no probability of any payoff so yeah my best guess at like a good way to approach this is yeah so certainly like trying to come up with a story for each given piece of basic research by which you might considering what you don't know what might you find out via that research that would shift the way things are currently done or yeah given technologies we you know if it's say fundamental physics research and you know we currently need technologies that solve such and such problem we can narrow it down at least somewhat what fundamental physics will provide us with new technologies or you know give the building blocks of future technologies that will solve those particular problems like we know the further work on yeah the basic components photovoltaics are more likely to give the solutions to to climate change then yeah research on perhaps the nature of black holes something even though in the long run like the black hole research might also be really really good yeah but it is very hard to kind of like sift out the things that have the you know to sift out the gems from the rest of like the things that have a particularly strong case of an enormous impact and it's largely going to be down to the experts in the particular field thinking through you know what they think they might find and then also figuring out what the implications of that finding would be. Do you think there's a risk that this kind of like quantitative calculations where you try to like actually estimate a number to compare with something else like if this relies on estimates that are very difficult to make that it becomes too sensitive to this so that it's will it always actually improve how good decisions we make or will it just give us like a number to back up a decision that is actually not of better quality than it would have been without the number 100% sure maybe there's certainly some risk of this especially if you know if people are abusing the MVP calculation or they're abusing the ITN framework likewise yeah if people you know if someone just wants to justify their pet research project and claims that oh it's a very high probability that I will in fact discover this thing despite not yet having researched that thing yeah it seems like if you go into this being kind of unbiased and using the best kind of evidential standards to figure out how likely it is that you would actually find something out or perhaps like deferring to your peers and asking them you know how likely do you think this thing appeared on this experiment and yeah also appealing to like other subject matter experts when you're trying to figure out the then implications of finding that thing so you know but the physicist is doing some research that might just ever so slightly might be the crucial breakthrough in like huge energy they should talk to non-physicists as well about what you know cheap almost freely available energy would do to the world yeah they shouldn't stick entirely to their own expertise but there's ways to mitigate this risk but there is still certainly a risk that you know people will make mistakes and miss out on some very good projects Issy do you think do you have an opinion or an intuition about like how applying these kind of methods would shift the focus between like very applied research or very fundamental research I want to answer that but can I go back to asking saying something going back to the question that you asked Hayden yes please do a thought on the matter and seeing what Hayden as the philosopher of the group thinks about the topic sorry to put you on the spot Hayden but you know if I think about you know this idea of what if in FIFA it dominates decisions it kind of is like reminiscent of this idea of like Pascal's mugging maybe or like the idea of these like high expected value low probability decisions starting to dominate all possible possibilities and how do we deal with that and it's reminds me I guess of like how people think about instrumental harm utilitarian frameworks and that's a different kind of problem to deal with but when we think about instrumental harm we're thinking about and Hayden led this like he's not advocating that in FIFA involves research that involves harming some to help many like and so and I think the reason that people typically don't endorse instrumental harm is people typically don't want to live in a society that has that as its norm and I guess I think about a similar idea around like risk preferences and so instrumental harm is a bit different than risk preferences when we're thinking about expected value but I guess the what I'm getting at here is like how we deal with the potential for these potentially super high impact expected value calculations that suggest basic research dominates all if that were to happen is to like essentially account for the diversity of risk preferences in society and ask ourselves how do we want to like account for the fact that not everybody wants our money to be spent on like super risky bets and so I'm curious like do you see though is that like a fair parallel to draw between instrumental harm and like risk preferences and in FIFA and expected value I think so especially also like one way in which it might not be is that even if we accept like expected value theory is the correct theory of rationality there's one question of you know what the entire you know community of scientists should choose as far as research topics and what I on the margin should choose given the current allocation of everyone else's time and effort and for the for the latter seems fine for me to take you know extreme long shots and do lots of basic research if it's the former and you've got a funding body kind of pop down deciding what everyone does you maybe don't want to put all your eggs into the basic research the kind of long shot basket so yeah like the appropriate theory of rationality might be different for those two different situations also expected value theory might just be false in general so perhaps the right way of calculating this isn't to won't result in everyone going for these extreme long shots and we can account for risk aversion and so on in like the usual ways that economists do when they're modifying expected value calculations to take account of that but that's complicated okay I can I know we're short on time I can try and answer your question Cecilia but you might have to say it again yes I was just asking if you would also expect like if a lot of research is starting to use these methods would you also expect as Haydn that this might lead to more of this kind of high risk high reward research or would you expect it to stay kind of the same or shift more towards this kind of safe bets research yeah I think it's tough to say I think there's still going to be a lot of people and it's just going to depend that people's risk preferences will vary even if they take a overall like welfareist approach and are using welfare as a criteria I still think they're I realistically don't anticipate there being like a strong shift in I think it's easier to sell people on wanting to improve people's lives and doing it a lot than selling them on the idea that they should also take the riskiest bets of all time no matter what and so therefore I think there's going to be a diversity there and you know what shifts and how much it shifts will probably be also dictated by you know the norms of funding agencies and incentives because as Haydn said we can always do what we want to do on the margin and we can make decisions but we do know that people's what they do will be somewhat guided by like what's professionally prudent and so I think like also people's behavior will be dictated by what is being valued at institutional levels thank you we are running out of time so I will hand over to folk to finish in a minute but I just want to give each of you a chance to say if there's anything you would wish for to happen in this field in the next couple of years do you have a new wish list? I would very much wish that both individual researchers as well as research funders came to prioritize research based on the social value and take the opportunities that research has to make the world a better place into account when deciding what research to engage in this research to advocate for which research to fund. I think we really have a great opportunity as researchers to make the world a better place and I would like to encourage everyone to believe that we really can make a difference if we try and it's worth trying to engage in these potentially more difficult less modern paths to do impact for research and that is very much worth the cost and I would like to see a shift in the direction of taking the social value of research into account at all of these levels. Thank you. Maybe hard to follow up on that one. I don't know if Essie or Haydn has anything to add. I would second almost all of that but I think I'm more skeptical than Falk of funding bodies or at least centralized government funding bodies doing a good job of this. Yeah I think I also third what Fox said and I guess I maybe I don't know if I'm skeptical like Haydn but I do think that kind of as I was saying in my previous comment so much starts at the top and we know that we can tell anybody and anybody listening to this can make and change individual decisions that they want but I do believe that the most promoting the most impactful research our symposium is great sure but it will be most impactful when funding bodies and institutional incentives align with that goal so my hope is that down the line or that's able to change Thank you so I'll hand back to Falk to finish we had the request also to get the links posted in the chat I hope you Falk are able to do that and somebody's asking if the recording will be available somewhere online and the answer is yes but I don't know exactly where Falk you are still muted sorry yeah the recordings will be posted on the website of the symposium and I'd like to thank all of you for attending the symposium and participating in the discussions asking your questions speaking along and if you'd like to learn more here's some links that you can check out to learn more about our research prioritization project if you'd like to collaborate on the project or if you have any questions or suggestions please don't hesitate to reach out this is a very open collaborative project there are many opportunities for you to participate in any way that that may suit you and finally if you'd like to try out some of the ideas that you have seen in the first two talks in the cross mapping and the important interactiveness framework on your own research topics then you can check out this spreadsheet linked here and just put these ideas into practice and thinking about what you want to work on in the future and I put all of these links in the chat so that you can follow them and check them out thank you very much for attending