 Thank you for joining us today on our new session as part of the GUESI webinar series. As you know, or for those joining us for the first time, the GUESI webinar series serves as a platform for the GUESI network members to share experience and expertise and evidence synthesis and to learn about potential collaborations. My name is Tamara Lutri. I'm the coordinator for the GUESI secretariat and I'll be moderating the session today. Before we begin, if you're using GoToWebinar for the first time, as you might have noticed, you are automatically muted as soon as you've signed up. So if you would like to submit any comment or question, please use the tab on the right. If you'd like to make the comment and address our presenter directly, you can use the raise your hand button and then we can unmute you so that you can discuss directly with our presenter. Our session is part of our webinar series on stakeholder engagement and environmental evidence synthesis. Our session today is on a five-step approach for stakeholder engagement and prioritization and planning of environmental evidence synthesis. And our presenter is Brianna Makura. Brianna is a colleague of ours, a friend of ours, and we're very happy to have her with us today. She is a research fellow at Stockholm Environmental Institute and environmental social scientists with the main research focus on the provision of robust evidence for decision making in environmental policy and practice. Brianna is conducting systematic evidence synthesis in the field of environmental management and working on the improvement of systematic review methods in the same field. She is a co-creator of ROSES, which stands for Reporting Standards for Systematic Use in Environmental Field and Editorial Monitor of the Collaboration for Environmental Evidence Journal Environmental Evidence. Thank you, Brianna, for joining us today. Thank you, Dan. So in a few seconds, we'll be able to see your slides. Yes. Yes, looks great. Perfect. So should I start with a talk? Yes. Would you like us to launch the poll first? No, I will let you know. OK, great. OK. So sorry about this. So hello everyone. As thanks Tamara for the introduction, as Tamara already mentioned, I will today hopefully give you an interesting overview of an approach that we were using in a project I was involved a couple of years ago in. And this is approach for stakeholder engagement in prioritization and planning of environmental evidence synthesis. OK, so this is just an overview of how this webinar will look like. First, I will give you a little bit of introduction to stakeholder engagement. Then I will try to introduce stakeholder engagement approach and give you some examples. Then I will share the challenges, the lessons we learned while applying this approach and also provide some recommendations. And finally, hopefully we will have some time at the very end to discuss and clarify things that you would like to know more about the approach I'm going to present. OK, so first to get us into the right kind of thinking, I would like you to answer a poll question and I would like you to tell me whether you have ever attempted to co-create or co-design your research with so-called non-experts. I am interested to know how many people have been engaged in co-creation or co-design processes. OK, so it seems that this webinar is going to be very useful because none of you seem to be engaging in any of the co-creation approaches. So that is great. OK, so let's move on. As you know, systematic reviews and systematic maps are golden standard synthesis methods that try to kind of increase transparency and comprehensiveness and try to minimize the biases during every stage of the review process. These review stages are identification and formulation of the review question, then publishing the review protocol, searching for studies, inclusion of relevant studies, assessing the validity of the studies if you're conducting a systematic review, if you're conducting a systematic map that step is not done, then extraction and synthesis of the findings, then finally writing a review report. However, in order to create a valuable reviews and valuable findings from those reviews for policy and practice, you need to ask a very relevant review questions. This is probably obvious, but also the findings of the review process. The findings of the review need to be recognized as the legitimate evidence. In order to do that, I would argue that engagement with organizations and individuals who use your reviews or affected by the reviews finding has to be done and it's very, very crucial, the engagement process. However, the stakeholder engagement process can be done throughout the whole review process from question formulation to report writing. But today's session and today's focus is actually only on the stakeholder engagement at early stages of the review planning process, at the question formulation and until drafting of the systematic review protocol. Okay, so first of all, I would like us to be on the same page and we'd like to share some definitions of the key concepts. I'm going to use throughout this presentation. So by stakeholder, in this presentation, I mean those who use or may be affected by a review findings and these stakeholders can be researchers, the subject experts, can be practitioners or people who are funding the reviews. They can be representatives of academia, governmental and non-governmental organizations, etc. So this is a very broad, as you can see, definition of a stakeholder. When it comes to the stakeholder engagement, I would like to define it as a bi-directional relationship between the stakeholder and the researcher that results in informed decision-making about certain stages of the review process and the review findings. Okay, so I will present your approach for early stakeholder engagement, but I will present that approach in a case example. As I already mentioned, this is a project that I've been involved in and this project is called Evidence-Based Environmental Management or EVM and it's funded by Swedish Research Strategic Environmental Funding. It was funded from 2012 to 2018, so the project has finished, but more information can be found in the link provided. It was, EVM was conducting reviews relevant, but not restricted to the Swedish context and these reviews were about environmental management and the aim of... Viljana, we can't... ...finally, yes? Yeah, we can now again. So can you please repeat the slide? This slide. Yes, so I'm going to show the approach on a case example of EVM, which is a project that was running from 2012 to 2018 and was financed by the Strategic Environmental Council or Foundation. It's called MISTRA. More information is provided on the slide here. So EVM was conducting reviews relevant, but not restricted to the Swedish context and aim was to improve the basis for decisions in Swedish environmental policy and management and it was composed of Executive Committee, methodology expert and was working with an international team of scientific experts. This setup was giving financial and political independence of this body and these reviews that were produced were reviews that can be considered as a public goods because they were produced with a public or general audience in mind. They were produced in an open access and they were not produced just because one party was requesting that review but was produced in a response to knowledge needs of multiple stakeholders. Okay, so how we were going about systematic reviews in this project and how was this combined with a stakeholder engagement? Well, every systematic review project would start with the implication of knowledge needs by very diverse and large group of stakeholders. These knowledge needs would then be prioritized and framed as review questions that would further be specified and made into systematic review questions. Systematic review would then be conducted by independent review team that was composed of the methodology and subject experts and then once the draft report was produced, stakeholders would be invited again to comment on that draft. Finally, final report would be produced with the final findings and these findings would then feed into the decision support tools. So this is the general context into which this stakeholder engagement approach was applied. Okay, so let's go to the approach to early stakeholder engagement itself. So this approach was done in order to incorporate stakeholder views and opinions in review planning and design. So the first setting up of the review stage. It was composed of the five steps. So identification of the stakeholders and then identification of the policy and practice relevance topics, relevant topics, knowledge needs of stakeholders, then framing and prioritization of the review questions, then establishment of specific scope of a review and then finally would finish with a public review of a draft review protocol. Okay, so this approach is published in 2007 in a special issue on stakeholder engagement and it's available in the link that is presented on this slide. Okay, so remember our evidence synthesis pathway from the beginning. Well, we are now focusing, we are now applying the stakeholder engagement approach just on this first bit of systematic review planning, as I already said. Okay, so how do we start with stakeholder engagement? Obviously, this stakeholder, every stakeholder engagement process starts with the identification of the relevant stakeholders that are going to get involved into the engagement process. This process is the most crucial process because depending on the stakeholders you select, you might have one or different types of the questions you end up reviewing, right? So this identification of relevant stakeholders has to be done very carefully, has to be planned very carefully and it's usually in this case done as a two-stage process. First stage of the identification of the stakeholders is done at the very beginning for the knowledge needs assessment stage. This stakeholder identification is done through the stakeholder analysis. There are many, many methods out there for stakeholder analysis, but this is what stakeholder analysis is about. It's actually about creating a list of stakeholders, just brainstorming, a list of stakeholders that might be affected by your review, that might be involved in your review, that might provide questions for your reviews. So this can include county and municipality administrators, other governmental agencies, funding agencies, NGOs, industry representatives, policymakers at any level. You have to think about a diverse set of stakeholders. Then you would normally describe the roles and responsibilities of each of the stakeholders and try to understand the level of their influence on your review project and how they are linked to the review projects and what are your expectations from them. Based on this, you would then prioritize the stakeholders as high priority, medium priority or low priority, and then decide to engage with a smaller or larger group of the stakeholders. The output of this whole process is a prioritized list of the stakeholders who may describe their knowledge, who may be relevant for you to describe their knowledge needs and propose review topics. Okay, so once we identified relevant stakeholders, we can then invite them for a meeting and we can engage with them and ask them for the needs, for which kind of knowledge needs they have in order to make informed decisions in this case, environmental policy and practice. So we would collect from them policy and practice relevant topics and their concerns that they might have and that might prevent them to make informed decisions in the jobs they are doing and that would be decision making in policy and practice. So as I said, this would be an interaction with a broad range of stakeholders across different sectors and the output of this process would be a list of broad, very broad stakeholder-generated topics and questions. The more detail of this process is described in the paper. And so these topics could be very global, national, all regional, environmental issues, perceived gaps in the evidence-based or simply controversial questions that were, for example, recently discussed in public debates. And you need to understand whether these topics are actually the synthesis gaps or places where there is enough evidence to connect to conduct systematic reviews. Primary search gaps where actually there is no any primary search on certain issue and needs to be, more primary search needs to be done or simply maybe these are translation gaps, a gaps that occur in science policy space where, for example, the knowledge from science is not translated into the policy. So these are all three different types of gaps or knowledge needs that the stakeholders can identify at this stage. So, for example, one of these broad concerns that were brought up by stakeholders during these stakeholder meetings is, for example, what are the reasons for the decline of seabirds in the Baltic Sea region? This is a very broad open-frame question. And if you're conducting systematic reviews, you will know that this question is not specific enough or not reviewable at all. However, this is just the first step in the process. Okay, so once you connect or collect the knowledge needs, then you would have to prioritize and frame better these knowledge needs into reviewable questions. So first you would conduct initial screening of the post topics and a post question at this stage whether these topics are reviewable at all. Is there a need to split them up or narrow them down? And after that, we would conduct a scoping studies of prioritized review topics. And a scoping study is a kind of a quick and dirty overview of the evidence base on a proposed review topic that tries to understand if there are any other systematic or traditional reviews on the topic. Is there sufficient scientific literature on the topic? Is there a need at all for a review on this topic? Are proposed questions at all scientifically meaningful? Are they answerable, conceptually clear, methodologically feasible, etc. But also you would in the scoping study kind of identify stakeholders that would be relevant for that specific review topic. Okay, so this is example of a scoping study that was done in order to understand whether biological control by Basileus of mosquitoes is working or not. Or does it have any effects on targets and non-target organisms and humans? And for example, some of the conclusions of this scoping study was that there is a fairly large number of studies on this topic of biological control of mosquitoes. But there are no reports on occurrence of resistance to BTI and that there is a fairly large number of studies on the effectiveness of BTI treatment in terms of mosquito abundance. And this scoping study proposes that a review may be conducted on this area of effectiveness of BTI treatment and persistence of BTI treatment in the environment. Okay, the other example of scoping study would be the one that was conducted in 2013 on the effects of plastic particles on growth and mortality of marine organisms, for example. This scoping study concluded that there are fewer studies that, only few studies investigated the effects of the exposures to plastic and that there is not maybe at that point of time was not a good moment to conduct a systematic review on the topic. Okay, as I said, apart from conducting scoping study, you would also and kind of identify review topics out of this general knowledge needs that were collected by stakeholders. You would also need to understand what are the stakeholders that will be most interested in this specific review topic. And so in order to do that, we would start from this initial broad stakeholder list and then kind of apply a snowball sampling in order to get more specific names and specific stakeholders that are relevant for proposed systematic review topic or area. And a snowball sampling, however, may entail community bias or overrepresentation of certain stakeholders in their interest. So it's always very recommended to conduct active searches for stakeholders with different or opposing interest in order to kind of combat the community bias. So the output of this process will be a list of stakeholders that are relevant for specific review topic and that can comment on a review scope in the next phase. Okay, so to summarize prioritization of the review question, would entail a scoping study, a list of prioritized review questions that coming out of that scoping study, and a list of review specific stakeholders. If you have several review questions proposed, as was an example in the scoping study on mosquitoes, you would invite key stakeholders to help be prioritized between different review questions. Based on the scoping study and additional stakeholder prioritization, in our case, Evian would invite an independent body, which is an executive committee to make a final decision on the review question. Because obviously time and resources are limited, so additional prioritization and this final decision need to be made by an independent body external to the review team and stakeholders. Okay, so once you decide for a review topic, this review topic might still not be relevant for all the stakeholders, might still be very broad, unclear, etc. So then you would invite stakeholders, again, to a review specific meeting. And this meeting would be led by the methodology expert and a scientist who will be the topic expert in the field, and they would seek, they would present a kind of preliminary plan for the review to stakeholders and seek input of the stakeholders to the scope and the focus of the review. So the stakeholders would be invited to focus on the PICO and PECO structure, on the inclusion strategy, on the search strategy proposed, even on the sources of relevant grey literature. And so this is kind of a way of narrowing even down the review question, but also still making it relevant for stakeholders. Finally, as I said, this is about fine-tuning kind of of the scope according to the stakeholder priorities. However, this has to be done with the methodological and scientific limitations and kind of questions of time and resources available and what is possible to do in terms of science and methodology behind. So the review team has to make a final decision on the review scope. Obviously, justification for the final decisions on the scope are communicated back to the stakeholders and are provided in the protocol. So the whole process is very transparent and no one feels excluded if the justifications are made in a very clear way. Okay. So for example, a review on the ability of wetlands to remove nutrients from water that was now published in 2016. During this final stakeholder engagement review specific meeting, during that meeting, the review scope was extended to cover removal, for example, of phosphorus from water and not only nitrogen, because that was more relevant to stakeholders, but also stakeholders expressed the need to, for example, focus only on creating and restored wetlands, but to exclude natural wetlands from the review scope. Okay. So then a review protocol, once all these comments are collected from this engagement process, review protocol is being drafted by a review team, by methodology experts, by subject experts, and then this draft protocol is then open for a public review. Everyone is kind of invited to comment on this draft protocol and obviously you have to put the time limit on this public review. So in our case it lasted two to three weeks and then these comments are incorporated in the final draft, in the final protocol and this protocol is being published. Independent body, in this case executive committee of our project would review the protocol development and stakeholder engagement process and finally approve the final changes to the protocol. And this early state with this early stakeholder engagement basically ends. Okay. In one of the examples or example reviews on roadside management, roadside management review was initially to include management effects on vascular plants and all kinds of animals. However, during the open consultation stage, the scope was narrowed down to cover vascular plants and invertebrates only and exclude, for example, mammals and birds because one of the stakeholders clarified that mammals and birds would not be good indicators of the impact of the roadside management. Okay. So now I would like to share with you some of the main challenges, lessons and recommendations from this whole process and this whole experience. Okay. So since we wanted to create reviews that are relevant for a broader audience, we needed to include the broad groups of stakeholders who then comment and affect the scope of the review. However, this is obviously time and resource demanding. This although may be in this presentation in this talk may be represented as a linear process. This stakeholder engagement is a very iterative non-linear process. It goes back and forth between the reviewer and the stakeholder group. This might be perceived as a challenge, but this also can be beneficial, obviously, to the review team. Also, what we notice is that sometimes it's very difficult actually to respond to knowledge needs, desires and expectations of the stakeholders and conceal these needs with established review methods. So you still have to keep the systematic review rigor in the methods, but you have to balance that with the stakeholder desires and expectations and this can prove difficult. However, this was helped by having ultimate decision-making as the executive committee, as an ultimate decision maker who made a final decision throughout the stakeholder engagement process on, for example, review topics or review scope and put the final say on where the reviews could go and which topics could they should address. Okay. However, it is very, very important to engage with the representative, diverse and well-balanced group of stakeholders. Like this, various biases and Western interests would be avoided. If you are searching for typical groups of your stakeholders, always search for a range of different views within these typical groups. So if you invite NGOs, there should be a different kind of types of NGOs in that stakeholder group. It's, if you are, for us, was quite easy because we were focusing only on the Swedish context and environmental management actors in Swedish context are very few. However, it's worth exploring the participation tool to kind of combat geographical bias and kind of spread the audience of, included in this engagement process. You have to remember that reviewers gain from stakeholders and this is a bi-directional exchange. Engagement with stakeholders, for example, help me and my colleagues to understand better the potential impact of the review for those concerned or affected by review process or review findings. Without stakeholder engagement, especially this very early stage, we were unable to kind of think or predict of potential impact and we would not be able to create questions that we would, we would, we are, we were more able to create questions that would be more kind of perceptive of different values and different views because, as you know, it depends on which question you ask, you get certain answer. And the most importantly is that I believe that early stakeholder engagement can facilitate endorsement of the review process. So the findings of the review are perceived as legitimate if you involve stakeholders at the early stage and keep them involved throughout the review process. Stakeholders need to feel that they participated actively and have opportunity to influence. So it's very important to understand that not everyone would be willing, for example, to talk in a meeting where everyone else is presented. So it's important to be aware of this kind of a power relations in a group of stakeholders and allow everyone to speak or to provide comments either in oral or a written form. And in order to increase this feeling of the participation and influence to the review process, it's also very, very important to stay transparent throughout the whole process so everyone would not feel excluded. And this actually will at the end pay at the end of the review process can pay better because then you also have stakeholders to which a group of established stakeholders to whom you can communicate your findings and hopefully hope that these findings will be taken up to further up to decision making process because these decision makers have been involved in production or production of the evidence they are going to use or they at least influence the production of the evidence that they are going to use in their decision making. Okay, so here with this I will slowly finish. Here are some references that I used in the presentation but more is available in that paper, the link to which you have in a couple of first slides. And with this I would like to thank you and maybe open floor for questions and further discussion. Thank you very much. Thank you Juliana for the very interesting session and definitely very informative. So we're going to start taking questions. Let's see we have a question from Sumant Kumbagiri Nagraj. Sumant I'm going to unmute you if you'd like to direct your question or comment to Juliana. Hello Sumant. Hi Juliana. Hello. Hello, yes. Am I audible? Yes. Yeah, it's a nice presentation. I'm from India working presently in Malaysia and you talk about including diverse stakeholders. I have a question. If the diverse stakeholders talk different language and understand different languages and there is no common language, do you have any suggestion? How do we include them in the research? Hi Sumant. That's a very, very interesting and valuable question and a point. In our case we haven't had a language barrier when it comes to stakeholder engagement but I assume that then engagement would have to be done through some kind of maybe not through a face-to-face meetings but through a kind of a e-participation where you would have to translate your questions that would be kind of posed to different types of stakeholders with different talking different languages. So perhaps this e-participation platforms where the engagement can be done in a virtual space in different languages facilitated by a review expert that and help with the translator I assume would be the better approach. This approach that we had so far was mostly based on kind of a face-to-face meeting where the exchange happens in a face-to-face kind of a discussion but I assume if there is a broader audience or audience that is more diverse e-participation would be one of the options for that. I don't know maybe others have some comments to that or some other ideas. Yeah I agree with your suggestion about the e-participation but if we introduce e-participation are we not limiting the participants only to be of a certain category who are internet literate or who have the capabilities to use the internet? What about people who don't have internet facilities who are not computer literate? Yes that's a very very good one. Yeah are we not excluding them by telling that we will include in the e-platform? Yes we do definitely so you would then include some kind of bias in your sample of stakeholders so I guess I don't know you would have to have translators in your yeah in your meetings this this seems to be like a like a only solution. I'm sorry that I can't provide any more clever. No no thank you thank you. This is actually a very interesting comment and we faced that with many of interventions or activities that are that are based actually on the internet but so Sumans I would actually like to ask you how would you suggest dealing with this or what would be an alternative that would not be restricted of actually that would not be restricted by specific access recommendations or requests? I think definitely as a suggestion came that we need to hire a translator and I would rather go for an interview. I would select my stakeholders and I would interview them probably what is their opinion and I would collect the findings. I don't see any other way to get the information from different strata of the society. Yeah thank you that's very interesting yeah definitely okay so thank you Sumans for your input and for this discussion. We still have a few minutes so if anyone else would like to make a comment or ask a question discuss directly with Rihanna please feel free to send your question now or raise your hand so that we unmute you. Tamara maybe I can ask one very general question. Yes definitely go ahead. I wonder if and thanks Sumit for starting this but does anyone else see any other challenges of the presented report presented approach in your working context? I would be very much interested to hear more about that since this approach was really really applied only to one very specific context. I guess we don't have any answers to this maybe it's a little bit premature to ask as people have not. Yeah maybe it's something thinking about after this webinar. Yes once people digest what they heard because it was a little bit too much of information. Yes potentially but great information. Yes okay so Rihanna thank you so much for this great presentation and I thank our attendees for joining us today please make sure to follow us for the other sessions for this webinar series. If you have any questions please feel free to send to Rihanna or send us the question we can direct to and connect you with Rihanna. Thank you very much once again everyone and thanks Tamara for hosting this webinar. Okay thank you so much Rihanna. Take care and health.