 Hello, everyone. Thank you for joining us today on this new GACI webinar series. So to remind you, the GACI webinar series serves as a platform for the GACI network members to share experience and expertise and evidence synthesis and to learn about potential collaborations. Before we start, let us go through the details of go-to webinar in case this is your first time. OK, so if you have any questions or if you would like to participate in the discussion, please use the toolbar on your right to submit your question and questions toolbar. And feel free to use the raise your hand button if you'd like to make a comment directly. So to start with introducing the webinar series, the stakeholder engagement webinar series will begin today for at least six months. And we'll be hosting one session per month. If you have any contribution that you would like to make, please feel free to contact us so that we see how you can add your input to the sessions. Secular engagement is an integral part of all systematic reviews to some degree. However, there has been a little discussion of this important process and systematic review guidance to date, particularly in the field of environmental management and conservation. The series discusses various aspects of engaging with stakeholders, describing the ranges of methods available, outlining experiences from various systematic review experts, and discussing issues relating to conflict, the benefits of training, engaging direct view decision makers, and communicating review results. For this first session, I'm happy to have Dr. Nien had a way with us to talk about a framework for stakeholder engagement during systematic reviews and maps and environmental management. Nien works at the Stockholm Environmental Institute SEI in Stockholm as a research fellow specializing in evidence synthesis. He also leads the Collaboration for Environmental Evidence CE Center at SEI. Nien's review work focuses on environmental management and international development. More specifically, he has worked on reviews in conservation science, human well-being, climate change, and greenhouse gas emissions, forestry, energy transitions, pollution, and multi-dimensional poverty. Nien is very active in publishing with the logical guidance related to systematic review methods, and is involved in a number of working groups with CEI. He has over seven years' experience of working with evidence synthesis and delivering training and review methods, and he is endorsed by CEI as a systematic review trainer. It's an honor for us for having you with us today, Nien. Thanks very much, Nien. Sorry, I just want to meet that. Hopefully everyone can hear me, OK? Yes, great. Thank you. OK, so hopefully everyone can see my screen. Thanks very much for that introduction, Tamara. It's really an honor to be here and talking to you today about stakeholder engagement. Like I said, like Tamara said, this is the beginning of a series of webinars about stakeholder engagement, particularly in environmental evidence synthesis, but really relevant to any sector. There's some really interesting presentations that will be coming up, so please do stick around and see what will be coming up over the next six months or so. As Tamara said, I'm a research fellow at CEI, based in Stockholm in Sweden. I'm also a research associate at the Africa Center for Evidence at the University of Johannesburg, and I lead the CEI CE Center at CEI. So this series really came about from a discussion that we had within the Collaboration for Environmental Evidence Stakeholder Engagement methods group, where we identified a real need for advice and guidance in how to conduct and report stakeholder engagement within CEI systematic reviews, but more broadly across systematic reviews more generally. And at the CEI conference in 2016 in Sweden, we had a thematic session where a number of people presented on different aspects of stakeholder engagement, and there we really identified this need for more guidance and better reporting. So some of the outputs of these discussions were an ongoing special series in the general environmental evidence on stakeholder engagement, and this ongoing series has been turned into an open access book, the first edition of which was produced in April 2018, and it's available on the EVM website, the link at the end of this presentation. And really importantly, a community of factors of people who are interested in stakeholder engagement, different aspects of the process and how we can integrate it into evidence synthesis. These are the titles of the papers in the special series so far, ranging across the whole process of systematic review production from question formulation, dealing with conflict, really thinking about what we mean by rigor in systematic reviews, thinking about training in systematic reviews as well in capacity building, approaches to prioritization of questions, lessons for introducing stakeholders to evidence synthesis, transdisciplinary working, and how to think transdisciplinary, if that's a word, thinking inclusively about involving people in prioritizing a multitude of questions in a controversial field of regulatory science, specifically GMO, genetically modified organisms. And then one of the most popular papers of the series so far that's been spread really across a number of ranges of different disciplines is this paper on storytelling. And we're thinking about how we communicate to tell a story, not being afraid of anecdotes when they're backed up by evidence. And then a final paper that came after the editorial that Sally Crow and I wrote that's talking specifically about engaging policymakers in systematic reviews with experience from DEFRA in the UK. So what we're going to be talking about today in this session is the framework for stakeholder engagement that sets out a suite of different processes and methods that can be used in systematic review to make the most of stakeholder engagement. So here's the paper that you can read. What I'm going to be doing is just giving an overview really of the contents of this paper. It's a very information-dense paper. So if you do have any questions arising after this or you want to learn more, then please do go and have a look at that. It's an extensive literature review and the results of a survey of systematic review practitioners as you'll see as we go through. But it really aims to be a kind of go-to resource when you have questions about systematic reviews and stakeholder engagement. So in this presentation, I'll be giving you a brief overview of what we mean by the term stakeholder and why it's important to engage with stakeholders as well. I'll then talk about how you go about identifying and selecting stakeholders or how you might go about doing that. And another process that really helps stakeholder engagement, which is analysis and mapping of stakeholders. And then I'll talk about two important concepts, achieving balance in stakeholder engagement and phasing engagement before talking about some of the challenges of stakeholder engagement and concluding remarks. So to start off with just thinking about defining stakeholders, probably the most common definition of stakeholders is from Freeman in 1984, which is any group or individual who's affected by or can affect the achievement of an organization's objectives. That's necessarily very broad to encompass anyone or anything who is involved or could be affected by an action. And it's difficult, perhaps, to think about how that would be applied in context of systematic reviews. So we conducted a survey of systematic review experts. And the definitions that we came across were things like people who are either affected by an issue or those who may be able to influence an issue, which includes local people, producers, NGOs, and governments. Some people said that it was anyone with an interest in a particular issue or anyone likely to be affected by an issue or a decision, which includes poor people, researchers, research experts, particularly systematic review methodology experts. People that have an interest in the subject matter includes researchers and experts, those generating evidence and the end users of the evidence, also include subjects of conservation and development projects. So this group of people that we interviewed was from a range of different disciplines, from health, development, and environment. Someone else thought it could be a personal representative of an organization that's affected by an activity that's being reviewed in one way or another and includes scientists. Those who have a stake in the question, policymakers, academics, educators, NGOs. Someone who has a stake in a finding is the issue you have real meaning in their lives, someone affected by the review findings. And those in one way or another that use the information from a systematic review, mainly those in decision making, for example, ministries, agencies on all levels, local, national, international, and includes scientists. So you can see there's a really big range of different definitions there. They're all broadly talking about the same thing, but some have quite a narrow definition, specifically, for example, people in decision making. Some have a broader definition that also includes people who undertake a review. But it's worth kind of unpacking that a little bit and thinking about why these definitions differ and why some people have a much broader definition than just the end user. And in thinking about why a broad definition is useful, there are a number of main reasons. One is that systematic reviews are often published as public goods. So we have the intent that they will be used widely and more widely than just the person who we perhaps were intending on doing the review for. So it's not often only the end user that might have asked for the review or we might really be targeting with the review. We make it more publicly available. So thinking more broadly about the definition of stakeholders helps us to anticipate the kind of people who might end up using it even if they weren't our intended audience. Another reason for having a broad definition is that we're then less likely to exclude marginalised groups. So if we have a broad definition and consider potentially anyone who's affected by the issue, then we might be less likely to forget to identify or just miss someone who might be marginalised by an issue. It also means that we are likely to have better planning and more resilience in our review if we think more broadly and flag up potential issues that we might otherwise miss if we only look at one specific end user. It also helps to identify and mitigate risk of unforeseen bias. For example, if there are conflicts of interest within the review team or the advisory group or the immediate group of stakeholders who are being actively engaged, we can help to flag that up and try to mitigate it before it causes problems. Another reason is that actors can help multiple roles and perform multiple actions as a stakeholder. So you might have an individual representing an organisation, but it's very difficult for them in a very heated discussion. Perhaps the only represent their organisation, they might be representing themselves as well. So thinking broadly about both citizens and organisations or individuals and organisations can be important. And then finally, it's useful to have a broad definition because it helps us to think not so much just about who the stakeholders are, but more about how the engagement works and what engagement they're going to be having, not just about the label that we give them. There are some cases when we might want to be careful about the term stakeholder, we might not want to call our stakeholders stakeholders. And that's because, as you can see, stakeholder clearly hides a lot of detail, it lumps lots of groups together. It also often disguises the need for tailored engagement that some stakeholders will need engaging with in different ways. And if we just talk about stakeholder engagement all the time, we might be missing a need to engage with specific stakeholders at specific times and in specific ways. There are some cases where the term is also contentious. One example is with the Sami indigenous people living in Scandinavia who object to the term stakeholder. And that's because it's been used widely in a financial and rights perspective around land use and land ownership. And the Sami people reject the idea of land ownership, so they've been excluded from discussions about stakeholders because they don't identify themselves as land owners or believe in land ownership. It means that they've been left out. So in a project with Sami, for example, we would avoid the word stakeholder. So then it might be better to talk about stakeholder more generally as a process, but then when it comes to how we're going to do that, be specific, who are we going to target, what's their role, what are they getting out for the review, what are they giving to the review, and being a lot more specific. So thinking about why we engage with people, there are a number of reasons for why we engage with stakeholders in systematic reviews. One more obvious one perhaps is that there's a moral obligation if we're dealing with public issues and public funds, we have a moral obligation to engage with people, to ensure that they benefit from the project and have a say. It also has been shown that by engaging with stakeholders, researchers have more access to more knowledge, so they're able to get their hands on a much more diverse and comprehensive body of knowledge than if they just work in isolation. It also means that stakeholder engagement in a project is more likely to have public acceptance of a project output. It's also then more likely to be judged as being successful, and it also means that the projects that have stakeholder engagement are more likely to have a broader and more intensive and perhaps successful communication effort as well. And then finally, as a knock on effect, it's been shown that stakeholder engagement can have an increased impact on decision-making as well. So when we think about stakeholders, as I said that we perhaps shouldn't think about who, but part of the process of thinking beyond just the who we're engaging with starts off by thinking about the who to begin with, and that could be advocacy groups, it could be businesses, citizens, decision enforcers, decision makers, publishers, research funders and researchers, very broadly speaking, and those terms might be defined quite differently in your discipline. But then if we think about the kind of roles that they play, they might be editors or peer reviewers, they might endorse the review, they might hold access to evidence, they might fund evidence, they might publish evidence or synthesis, they might help to communicate research, they might ask questions, they might be reviewing research, they might help to influence the scope of projects, they might provide a service, for example, providing an intervention in the real world, as it were. They might use that service and they might use the review. And then beyond that, we can think about a range of different actions that those actors in those roles can have, and they could be suggesting sources of literature for the review, they could be submitting articles to the review, they could undertake the review themselves or perhaps endorse the review. They can facilitate access to the review, they could read the review themselves or perhaps read it and then share it. They could integrate the review findings into their decisions somehow. They could help to set the standards of the review in terms of its methodological rigor. They could provide funding for the review or in kind contribution. And they can share knowledge and experience for setting the scope and the context of the review. So if we think with this kind of conceptual model, it helps us to think about different people and how they engage with the review. So some examples could be a concerned citizen who uses a review on the impact of plastics on marine biota and they integrate the review findings into a decision about whether or not to purchase plastic water bottles. You could also think about a research council that might fund a review into the efficacy of crayfish conservation in the UK and they will provide money to the review and integrate the findings of evidence gaps into the funding of primary research. So it's worth thinking about stakeholders This is just one framework for thinking about them but thinking about them in a broader way. And this is just a schematic of the ways in which stakeholders can engage with a review by providing something or by the review providing something to them. And the schematic is provided in more detail in the, or described in more detail in the framework paper. So there are a range of different benefits of engaging with stakeholders. Some other, for example, it facilitates transparency by dealing with stakeholders you need to be transparent about the processes that you're using. And one concept that we use to describe this transparent approach is the glass box approach as opposed to a black box approach. And by being very transparent about the methods that we use within the review we're able to then engage with stakeholders in a way that we can describe our research and say the robust methods that we're using and they can ask questions and verify what we're doing. But we avoid any undue influence on the review's findings and the process that we use within the review. Another benefit is that you can help to predict controversies that might crop up around either the review's results or the review's methods. This is what was not done by Sanchez-Bio and Wickhoy I'm not quite sure how you pronounce that name in a review that was published earlier this year in biological conservation. And in that review, they had searched only for three search terms and only in one resource. And they clearly missed a large body of literature and they had a very biased review question. And if they had engaged with stakeholders at the beginning those kind of issues would have cropped up and they could have prepared answers and improved their methodology before starting their review. Stakeholder engagement also helps to ensure that you're using broadly accepted definitions and you don't end up trying to talk about a concept using terms that people define and understand differently at the end. It also helps to ensure that the methods that you're using are very rigorous particularly search strategy. That's particularly important if you're able to identify people who can tell you additional terms that you might have missed additional synonyms that might be used differently in different contexts. They can help to provide access to gray literature by showing you sources of gray literature that you can search and also providing access to reports that you might otherwise not find. And they can also endorse and help to accept the review. This is particularly the case if they're involved from the start and you have a kind of co-design approach that we'll talk about in a bit with the protocol because they'll feel a sense of ownership hopefully that they have been involved from the start. And this also helps to improve communication at the end. Stakeholders can be really useful in helping to tailor communications. You don't have to second guess the kind of information that decision makers or practitioners want. You can just ask them. Which kind of output would you like? A one-page fact sheet, a short video, a podcast and they can help you to tailor that communication to be in the right format. One additional thing is that they can help to document the impact of your review. So if you have stakeholders who you've engaged with and built a relationship with throughout the process of your review, it's then much easier to document and demonstrate that your review has had an impact, whether it's just being in the back of someone's mind when they make a decision or whether it's actually being used to change that decision or spark off a new decision system. And then a final benefit is that by engaging people with an evidence synthesis process, you can build capacity for evidence-informed decision making and build awareness around critical appraisal and evidence synthesis approaches, things like critical thinking. These are just some of the benefits as well. There are a range of different modes of stakeholder engagement from the informing passives side of things on the left over to the empowering side of things on the right. Research that tends to only inform in its state of engagement is typically research driven in terms of where the questions come from. The engagement is passive, it can be selective and it can be quite exclusive in terms of who it involves and how. And there's generally very limited or sometimes even no feeling of ownership of the research more broadly than the people who are involved in coming up with the question. And then through consulting people, involving people, collaborating and then empowerment moves over to a needs-driven approach that's very active in terms of its engagement, that tries to be comprehensive and as inclusive as possible and have a feeling of shared ownership. Typically systematic reviews as far as they can go would be around the collaborate and the start of the empowerment. And that's because the systematic review process once it started really sticks as closely as possible to a predetermined protocol. And the people who then conduct that research need to be experts, subjects and methodology experts. So co-conduct of a systematic review is not quite so common, but co-design by sitting people down and helping them to, or having them help you design the protocol is something that we would often do within a systematic review. So one of the first processes that you'll want to do in a systematic review is to identify which stakeholders to involve and to select them to be involved in your project. There are a number of different methods that can be used for identifying and selecting stakeholders. One of the most common is probably purpose of selection and that's the use of known contact with people who you're already aware of. One of the problems of this approach is that you can potentially end up with a bias sub-sample because you only cherry pick people that you're already aware of and it risks ignoring minorities. One of the benefits of this approach though is that it's easier to access and then maintain an engagement with people who you already know. It's also then easier to deal with stakeholders who are a smaller group, which is more likely what you're going to end up with from purpose of selection. Another approach, which is snowballing, is the process by which you ask key stakeholders to make suggestions of other stakeholders. This can also end up with a potentially biased sub-sample and does also risk ignoring minorities, but the risk of this is reduced if you use multiple starting points, if you ask multiple key stakeholders to suggest others and spread out the network that way. The benefits here are that unknown intermediate to media is more likely to end up with a response, a successful response from people that you invite. And also by, as I said, by having multiple iterations or multiple starting points, you're reducing the likelihood of ignoring minorities. Another process of identifying stakeholders by having an open call where you publicize the needs for stakeholder participation publicly. This does risk missing people who don't have an access to the advertisements you need to think carefully about where you put it. And you might end up with a potentially unmanageable large group of stakeholders who potentially misunderstand the aims of the engagement because the advert is made passively. You don't have the chance to chat to people about the involvement. And you do also risk swamping minorities in the groups that you end up with because you could have a very large group of very commonly represented people and minorities find it harder to then have their say. But one of the benefits is that some biases like identification bias and networking bias are avoided. If you want to know more about the biases that are listed and described in the paper, I won't go into too much detail about them. But you do end up with a potentially wider diversity of stakeholders by having an open call. And then a final method is a systematic search, much like a systematic search for evidence in a systematic review. You systematically search for relevant stakeholders, for example, by doing a search on Google or in a database of stakeholders. The problems here is that you end up with a much larger volume of stakeholders to engage and you either have to cut it down or engage with a very large group. Or you risk missing people with little online presence. The benefits are that it's less likely to be biased because it's systematic, but then that depends on the search and you use much like in a systematic review. And it's also repeatable and has a justifiable methodology. Typically, you use a range of different methods depending on the resources that you have available. Once you have identified your stakeholders, it's really important and useful to map them and to analyze them to try and have a bit of an understanding about who they are, what they want, how they can engage and the best way of engaging them. And stakeholder mapping is good because it helps to ensure a balance in the stakeholder groups. And we'll talk about this a bit. It helps to prioritize certain groups of stakeholders when you have limited resources in your review. It also helps to identify, flag up and mitigate potential conflicts between stakeholders, particularly if you're dealing with a highly contentious group. It also helps you to tailor the contact with your stakeholders so that you contact people in the right way rather than just inviting everybody to engage in one way. You try to understand the best ways of engaging with different people to ensure that how you engage is efficient. And it also helps you to phase that contact. So depending on the utility and the benefits to and from each stakeholder, you can phase them to start engaging a certain group of stakeholders perhaps and then change the group of stakeholders depending on what you're trying to get out of them and the benefit for them from the review. There's typically two different approaches to this stakeholder analysis or mapping approach. The most common approach is top-down approaches where the review team or an expert team perhaps the advisory group will classify stakeholders based on their knowledge of them. That may be specific stakeholders or it may be just groups of stakeholders like journalists rather than specific organizations or named people. And you might want to just think carefully about how transparent this process needs to be. We try to be as transparent as possible in systematic reviews and the process should be the same stakeholder engagement but we might not want to be very transparent about some of the discussions that we have about who we engage and when if we think some of that information might be sensitive. The other approach is a bottom-up approach where stakeholders classify one another. This is useful where there's a subject that's highly contentious or where there's conflict between stakeholders or where one of the key concerns in the project is its legitimacy but this kind of bottom-up approach is very resource-intensive. You need to spend a lot of time actively engaging with a large group of your stakeholders. So thinking practically about how you might do this stakeholder analysis process one of the most common ways of doing it is through matrices and the most common one is an interest influence matrix but you could change this up if you've got some different axes that you're interested in. So to explain you would classify each group of stakeholders according to two dimensions. One might be the level of interest how interested are they in the project and the other access might be access rather might be what influence do they have in supporting our goals how influential can they be. But you can do this across other dimensions for example the amount of evidence that people could provide versus the level of effort needed to engage them or the if you're thinking about communication the potential level of influence on social media versus the cost of engaging with those people. And what you do is you then classify them across these two axes. So here we have the interest power or interest influence matrix and you classify them as low medium or high perhaps and move them around actively in a brainstorming activity with your group. And broadly speaking you classify this into four groups where people have a low interest and a low power you might want to just monitor them they require minimum efforts where people have a high level of interest but a low level of power or influence in helping your review or helping having impact then you might just want to keep them informed. Where people have a low level of interest for the high amount of power you might either want to just keep them satisfied or try to increase that influence in the review and where people have a high interest and a high power you might want to manage them closely and have active engagement to make sure that they're well informed and the review benefits from them as much as possible. You can find out about this process very easily called stakeholder mapping or stakeholder analysis. We go into detail in our paper but you can also find more in the paper at the bottom of the slide there. So now move on to talk about balance, phasing and planning of the stakeholder engagement process. Firstly on balance what we mean by balance in stakeholder engagement is the representation of all main interests views and opinions. What we don't mean is quantitative or proportional representation so we're not trying to get an equal number of people from different groups or a proportional representation of different groups in proportion to how many people there are working in those different groups across the stakeholders more generally. And the process of trying to attain balance is so that we can allow all relevant groups and individuals to have their say and also to empower marginalized groups. The thing is balance is quite difficult to detect it's difficult to say that you have a balanced group. It's most evident when it's absent so if you're in a group and there's a clear imbalance you'll really feel it and you need to think about how to address that how to bring in marginalized groups or groups that aren't having their say. And it's worthwhile also just mentioning that we should consider social equity as well as conceptual or role balance. So as well as thinking about the different groups of stakeholders and roles that they have as individuals or groups it's worth thinking about aspects of social equity like gender and ethnic backgrounds. In terms of phasing stakeholder engagement I've mentioned this a bit already but it's going to be disadvantageous if we try to engage all stakeholders at every process throughout the systematic review the process of conducting a systematic review. So what we've done in our paper is to identify the kinds of engagement actions that stakeholders can have in a review when they happen during the review and the direction of the action whether it's benefiting the review or benefiting the stakeholders. And so you might want to think about this kind of framework to identify which of your stakeholders do you want to engage at which stage. You might have some that you engage all the way through the process but you might have some that you only engage at the start or you only engage at the end. Being transparent about that phasing might be something you want to consider as well. And one thing to consider is if you're trying to engage someone all the way through as you you should consider carefully if they are a named individual that over a two-year systematic review project that person might have changed roles. So you might want to build a relationship not only with the person but also with the organisation that you're interested in. When you're thinking about planning engagement there are a number of things you need to think about because they'll affect who you have in your pool of stakeholders and how they engage with you and how successful that engagement is. One is you need to think about how to invite people and you're going to have a closed call or an open call or you're going to use email, letters, phone calls or you can invite people in person or by having a public posting. How are you going to engage with them? Will you have group meetings? Will you have individual meetings? Will you contact them by telephone or by Skype or by email or send them a questionnaire that they will fill in? You might want to think about having different engagements for different actors that tailoring that contact partly because the contact is going to be more efficient and more successful. Perhaps the number of invitations that you get accepted will be higher if you give people different ways to engage rather than only inviting them to come to a physical meeting. There's aspects about access to transport costs or having their time paid but also issues around getting the same or two different groups of people in the same room if you've got a topic that's very conflicting. And then think about what you want to ask them. Are you asking them for support for endorsements for their comments and opinions or suggestions and being clear about that so they know exactly what you're asking. And as I said, it's already said already it's useful to know at what points you want to engage with them through the review and think carefully about why you want to only engage with certain stakeholders at certain points. Might they want to be involved earlier? So giving people the option of phase contacts rather than just deciding that for them is useful to consider as well. And then how you ask is also important. Thinking about the terminology and explanations that you use and try to avoid fatigue. Systematic review methodology is a very complex concept to explain to someone and perhaps if you're just talking about the research project maybe you don't need to explain that to someone who you're only engaging with at the communication stage. Maybe you just talk about the research project. And maybe you think carefully about having supporting information that explains exactly what a systematic review is but try to avoid bombarding people with complex terminology. And then as I said before it's really important to have clear objectives in terms of what you're asking them to do rather than just inviting them to come along for a day to discuss something. What are they going to be doing? What are they getting out of it? Were you getting out of it? What are you going to be doing with the information they give you? How might they help to influence research and what benefit might that have for them? So remember challenges with this approach you can probably tell already that it's quite complex and can be a very time consuming and laborious task. But I just wanted to take some time now to think about some of the challenges. Firstly, it's very easy for bias to creep into the stakeholder engagement process and you can end up with a biased stakeholder engagement. That doesn't necessarily mean that you'll have a bias systematic review and having a system in place that protects your systematic review from undue influence is important. You can do that, for example, by giving people the opportunity to comment but not necessarily taking all of their comments on board or having an advisory group agree whether stakeholder engagement comments come into influence the methodology. But just to talk briefly about some of the biases you won't be able to read this table but it's provided in more detail in the stakeholder engagement framework paper. But I just wanted to highlight a couple of as an example. One is at the stakeholder selection phase and this is called identification bias. And the purposeful selection of stakeholders using personal or organisational knowledge or unsystematic searches might result in a bias or unbalanced group of stakeholders. One of the ways to mitigate this is to use a combination of selection methods as I described before. Another example is during ongoing engagement in the review that you have something called commitment bias where stakeholders might not be able to commit to being involved along the whole systematic review process and that causes stakeholders to drop out over time which might leave you with an unbalanced or bias group of stakeholders despite trying to have maybe successfully having a balanced group at the start. One way to try to get around this is by fading contact with certain stakeholders according to their likely involvement so that you don't get them dropping out before they're actually having a chance to do something. But you can find more about these biases and the mitigation measures to avoid them in the paper as I said. So this is just a schematic that shows the main stages of a systematic review from a recent project that aimed to look at the time requirements of systematic reviews. And you can see that from that study the planning time which is the part that really has to staple their engagement for most people is a large proportion of the systematic review time unsurprisingly. But also the kind of administration time dealing with people developing the protocol dealing with stakeholders suggestions and getting back to them and also communicating the results of the review together take a huge proportion of time. So you need to plan it carefully you need to be efficient you need to be smart when you're doing a stakeholder engagement. But thinking about some more challenges as well many people think that good stakeholder engagement can divert resources away from the conduct of your review and to a large extent that's true. We all know that budgets for systematic reviews are typically quite small and it would be easy for a stakeholder engagement process itself to swallow almost an entire systematic review budget. You can build a stakeholder engagement process for restricted budgets by just planning carefully and thinking about which are the most important aspects to get right. As with a systematic review planning from the start helps to avoid unforeseen problems. It can also be difficult to maintain balance and representativeness particularly if you're trying to engage everyone all the way through the process. There's a need to manage stakeholder expectations as well that they might be expecting to have an impact on the review and you need to be clear that certain kinds of impact won't be possible shouldn't be possible. As I said you want to avoid overwhelming them with jargon or too much information. Equally you want to avoid undue influence on your systematic review from your stakeholders so being clear about the kinds of things that stakeholders can contribute to and the kinds of impacts that they can't in terms of the systematic review methodology and output is important. In many cases you might want to start off with a default of providing anonymity to your stakeholders if you do report your stakeholder activity which we'd encourage you to do but then you should think about giving acknowledgement and that stakeholders who've given up their time to turn up to a stakeholder meeting might really deserve to be acknowledged but then considering their anonymity is important as well so the balance between default anonymity but asking if they would mind having their names provided in an acknowledgement might be an approach you want to take. As always there's a potential for stakeholder conflict Annika Nilsson and Rasmus Larson from the Stock of Environment Institute written a paper on conflict and Annika I believe will be presenting on conflict later in the series. Planning how you manage conflict within your stakeholders and between your stakeholder and your review is important thinking about whether compromise might be possible and what that might look like. And maintaining long-term interest in your stakeholder is important as well. You should also think about avoiding tokenism so don't just do stakeholder engagement because you think it's important do it because you think it will be good for your review and do it well. You might end up needing to either bring in a specific stakeholder engagement expert or training your team in stakeholder engagement because it can be quite a complex process particularly if there is conflict management needed and ideally it's worthwhile trying to monitor and evaluate your stakeholder engagement activities to know particularly if you're going to institutionalize systematic reviews in the future what can you do to improve your stakeholder engagement what did you do right what didn't you do right it's also for the benefit of the larger systematic review broader systematic review community as well to learn what works and what doesn't in stakeholder engagement. And then just before we finish some final considerations firstly on communication that I'm sure you all know systematic review publication doesn't mean communication once we published our long systematic review it doesn't mean that it's been communicated to the right people and then we need to take the key messages and links to the evidence for those key messages out of our review and send them to our the right stakeholders and we need in that process to think about who are these messages for and what format is most likely to be effective in terms of communication. And those key contacts that we have can help to tailor those communications and test them. But communication is easier if the stakeholders are engaged throughout so by thinking of communication integral parts of stakeholder engagement and engaging stakeholders from the beginning we're much more likely to have a higher success in our communication it's much more likely to be impactful because stakeholders will feel a sense of ownership and then some final considerations as well we should balance the need for transparency in stakeholder engagement in reporting what we've done with engaging the stakeholders with this need for sensitivity and that relates to anonymity. We should be aware of balance and power in the group who are identifying and analyzing stakeholders so if we choose to go for a top-down approach in stakeholder identification and mapping then we should be careful about a potential for bias or balance or imbalance or an unpleasant power relationship going on there. We should also be reasonable and feasible about who we expect to engage. People are often being asked to engage in research projects particularly Indigenous groups and if we are all inviting them to get involved with all of our reviews then we might be expecting a bit too much but if we invite the CEO of an organization to take part is he really going or she really going to perhaps we should consider someone who might have less power but would be more likely to be engaged. Also this relates to the same point don't ask too much that many researchers are asking for input and we shouldn't ask we shouldn't all be asking the same people to be involved in a large number of reviews and then finally plan carefully check out the resources that I've mentioned and that we review in our paper and good luck. Thanks very much. Thank you so much Neil this was a very informative session and very happy to have started the series with this one so we're going to take questions now please either send us your question and the toolbar on the right or use the raise your hand button so we have one question on whether the slides will be available for attendees whether this slides will be available. Yeah I'll be posting the presentation on the OS Open Science Framework but I'll also try to make it available through Guessie. Great thank you so if we don't have any specific question at the moment I see that we have Richard Morley here with us from the Cochrane Consumer Network Cochrane Richard it would be nice to hear from you about your input on how this method for stakeholder engagement is how we can prepare it to the consumer engagement within the Cochrane method or whether at least the approach to targeting stakeholders is similar maybe. Hi Tamara thank you very much for inviting me to speak and thank you Neil for a really interesting presentation I'm aware of your work and I admire it very much so thanks very much for that Yeah so I'm Richard Morley and I'm the consumer engagement officer for Cochrane and so I work to support stakeholder involvement in Cochrane's systematic reviews well a particular subset of stakeholders which is consumers, patients, carers, service users however you want to describe them and I suppose I would just I can say something and then there's a question for Neil perhaps which is that if people are interested in stakeholder involvement in Cochrane reviews if there's a website which is consumers.cochrane.org and there there's a lot of information to support people who want to involve consumers we have a terrific resource for review authors review author teams which we've called involving people and that has a wide range of frameworks and downloadable resources to allow people to give people support in involving consumers in their work and also there are some papers which we've published which describe the way that we approached this work the whole thing was part of something called the active project and involved researchers and consumers and other people involved in Cochrane to produce these resources so we're very pleased with those and I think they're practical and useful for people I think so that was my just very quick advert Neil I wonder if you have any thoughts about this we when we presented the active resources we presented a range of different ways in which consumers in particular could be involved in research in evidence synthesis but we didn't we didn't draw any conclusions about the best way to involve people or the most effective ways and so on I wonder if you have any thoughts about that about what the evidence says about about you know yeah there's a range of different ways to involve people but what's the best way because I have to say that's a question that authors and editors in Cochrane often ask yeah you know we don't have a lot of time so what's the best way given the limited resources to involve people given that we have to do it that thanks very much that's a really good question Richard I I mean in terms of environment I know that there isn't enough evidence in terms of effectiveness of stakeholder engagement and I'm not that familiar with the health care evidence I know that you guys have leaps of bounds ahead of us in terms of stakeholder engagement more broadly yeah we we tried to also not speak about what's best for particularly that reason that I we weren't aware of evidence that suggested which method was best and there are pitfalls with all of them but I'm simply not aware actually I'm just not sure what what the evidence is in terms of best practices I do know that there are a couple of efforts to do systematic reviews of stakeholder engagement Alex Alex Pollock I think to this systematic review but that was more about reporting at least the one I saw was but I think it's related to a body of work that that group's interested in but I know that monitoring the valuation of stakeholder engagement is generally not done so it just highlights the importance to try to track what works and what doesn't I imagine the body of evidence is quite slim but I just don't know I'm very that's a really good question thanks yeah I mean Alex was one of the collaborates well the lead actually the work on the active projects and so I was very privileged to work with Alex yeah thank you thank you very much Neil thanks for your presentation which was very interesting very very clear thank you Thank you Richard for joining us today If we have a couple of questions I don't know if you have if we have time one is could you speak a bit more on empowerment in most stakeholder engagement and why does may not be feasible in systematic reviews I think thinking about trying to empower marginalised groups in stakeholder engagement or trying to avoid power imbalance is a really tricky issue I think one thing we should probably try not to do is try and fix power and balance in a system when what we're trying to do is stakeholder engagement for a systematic review because we simply won't have the resources available and we shouldn't try and do something without the appropriate resources but I do think there are ways in which what tries to be a very independent method as systematic review tries to take a back seat in terms of political views and predetermined ideas of what results might show I think systematic reviews are quite a useful tool for trying to demonstrate that if there is an imbalance between particularly between research and a certain group of stakeholders systematic reviews are trying to address that balance in terms of the level of independence and the procedural objectivity that they try to attain so I think by systematic reviews can be a useful tool to try and reduce conflict between research and certain groups of stakeholders but I think my main point would be avoid trying to fix power and balance in a system only as part of a stakeholder engagement process and then there's a question from Amina do we consider the systematic review team coordinating the review among the reviews stakeholders yes I think I really like the idea of this broad definition of stakeholders because conflicts of interest within the review team are particularly in healthcare right now are a really hot topic and by thinking more broadly about stakeholders we can try to be transparent and be open about the kind of conflicts that arise one conflict for example is that systematic reviewers are trying to gather a systematic review finished and if they run out of money or run out of time the rigor of the systematic review process might be challenged and so by being clear about that we can try to build in appropriate planning into our systematic review process that's kind of an obvious one but there are other ones where we can try to look for conflict across all stakeholders not think of ourselves when we do the reviewers being above or different to any other stakeholder and then is it necessary to distinguish between patients and users, consumers and the public I think that just depends on if you think that they would engage in a different way they would benefit from a different way or bring anything in a different way maybe you want to phase them differently I think that depends on your topic really and then finally is there a specific framework for categorizing systematic review stakeholders that you can recommend no I don't think that is because I think it's so disciplined and subject-specific that you probably want to do one yourself from scratch but I don't think that stakeholder engagement in systematic reviews is very well reported across the board but you might find a systematic review that does describe stakeholder engagement and they might have used a categorization framework that you would like to use and think is useful so that's probably not very useful but yeah I think we're getting better at this and our idea with our special series was that we wanted to try to get people to report stakeholder engagement in a more transparent way to help other people design stakeholder engagement so that's all the questions I think okay so thank you so much this was a wonderful session if anyone has any questions please feel free to contact me directly or you can send us an email and we can forward it the recording for this webinar would be made available within one week for free on the GSC website thank you and join us on the next sessions part of this webinar series goodbye now