 Okay so we're live now. Hello everyone, welcome to Esmar Conf 2023 and this workshop on screening studies for eligibility in evidence synthesis. My name is Bibiana Vetankur. This workshop is being live streamed to YouTube and has a group of participants also taking part live. A very warm welcome to all of you. If you have any questions for our presenter today, you can ask them by the ES Hackathon Twitter account by commenting on the tweet about this workshop. Also if you registered for the workshop, you can ask your question here on Zoom in the Q&A facility or you can open your mic and ask the presenter directly. You can also comment in the chat and chat with other participants on our dedicated Slack channel. They should have been sent along with your registration information as well. We will endeavour to answer all your questions as soon as possible. Also we would like to take time to draw your attention to the code of conduct. This is available at the Esmar Conf website at www.esmarconf.org. Our workshop presenter today is Jacqui Eales from the European Center for Environment and Human Health at the University of Exeter. She's also a research scientist at the Earthwatch Institute. Jacqui, over to you. Thanks Bibiana and a very warm welcome to everyone and I hope you enjoyed this workshop. So this is a workshop which is really aimed at people who don't necessarily have a full knowledge of the systematic review process, perhaps just starting a review relatively new or perhaps you just like to know more about the process itself. It can also be useful for people who have done this before and would like some tips. As we go along I'll be hopefully answering some of your questions and at this point it's quite a good time to say that please do interrupt me with questions. There are lots of options as Bibiana just said to ask those questions, lots of methods but for those of you in the meeting today please do feel free to unmute to put up your hands whichever you prefer to do and just to really interact with this session because I think it's a really good opportunity to ask those pressing questions. Now so as many of you know systematic processes for deciding which studies to include is a key stage in any evidence synthesis and this workshop's really going to present the key principles of screening which are to be transparent and objective with using some worked examples, some exercises and as I said lots of opportunities for those questions. So the importance of transparency we're going to cover that we'll also go through how to design eligibility criteria so what should you be included including in your eligibility criteria what should you not. Any of the common pitfalls that many people fall into and how we can avoid these and also what tools, softwares and new developments there might have been in the recent years. There are quite a few of these available to aid the screening process and these are just becoming more and more. A quick look on SR Toolbox will show you that there are plenty of softwares which are being developed all of which have their pros and cons and many of us have our favorites and are not so favorites so we'll be looking at some of those a bit later as well. And also this is a good opportunity for any of you who are working on some tools perhaps using maybe using text mining to help aid the screening process. It's an opportunity for you to talk about what you're doing as well in your work so please do feel free to unmute and pop things in the chats. But really I do want to make sure that you have covered the key questions which you have burning so what if I can't make a decision on a particular study I can't find a full text or how detailed should you go with eligibility criteria. Now these are some of the key questions which often come up when doing screening for systematic reviews and other evidence synthesis and we'd like to be able to cover those as we go along so please do shout out. So in the process of doing an evidence synthesis article screening or eligibility screening or study inclusion they often get called the same things is one of the most time consuming processes in conducting your evidence synthesis. So in some cases there can be tens of thousands of search results which need to be screened at the title or title and abstract stage leaving several thousands of abstracts to be screened and this can lead you to hundreds of full texts which need to be found retrieved read assessed and then carried through to the next stage. But we know that this stage this quick screening stage is really vital to ensure that all of the relevant evidence that we can find is included and that the really careful carefully curated balance of sensitivity and specificity in the search which was designed doesn't go to waste. So during this session we're going to discuss some techniques that might help us save time during screening as well. So the screening process now we have our search results which we've carefully curated and put into perhaps one library of references perhaps if you've got a lot of great literature it's not possible to do that you have to put it in manually there might be several different libraries you're having to look at. You've removed duplicates where possible you've checked the web you've checked organizational websites and you've retrieved all the great literature you can. You are now ready to start the screening process I'm perhaps I should say at this point also that it's not necessarily that this screening process is standalone it might be that it bleeds into other parts of the reviews. I'm doing a particular review at the moment where I have I'm doing a lot of snowballing which is where we're looking at the reference lists of existing articles to see if there are any more relevant ones so it's kind of like a snowballing technique or a pile growing technique sometimes and this is going on all the time every time that I'm including a study I need to go and look at the reference list of that study and do forwards and sometimes backwards citation chasing so this screening process can be really iterative and of course if you've done reviews before you might know that screening and testing screening or scoping is part of the developing the protocol stage so once you launch yourself into the full stage of screening it might be that you've already had some experience of doing it because you've already done some some screening in your protocol development. So the question we're trying to answer here is really how do we decide which of our studies are eligible for inclusion in this review well first of all it's quite useful to underline the importance of transparency now this of course carries throughout the whole of a systematic review process or evidence synthesis process and particularly in the screening stage it means that our methods are easy to understand and whilst we're going through that once we're undertaking the methods but also for people who are going back and reading those methods later we can prove that the methods were followed by being detailed and transparent about how they were undertaken. We can go back to particular articles you may have a situation where there might have been a so-called mistake made in screening where so an example is where an author has come to you and said well I know that you're doing this evidence synthesis in this topical area have you seen my paper and you look at the paper and you think oh that's really relevant well have we missed it somehow has it has it been not picked up in the search if it had been picked up in the search it would have gone through to screening and that's where it can be really useful to find out what happened to this particular article why was it screened out and that's where transparency can be really important so keeping really meticulous methods I can't underline that enough due to a painful person experience and it also helps updates to be undertaken updates to your evidence synthesis sometimes that's needed to be done before publication it can also be a lot more efficient by being transparent so you've often got many search results and by keeping proper records that's the only way to keep track of what's going on in your review by using a preset number of eligibility criteria across different stages of screening this can really help us be much more efficient and we'll talk a lot more about what those eligibility criteria might be later on and it also enables the full reporting decisions and these to be openly accessible for future works we try to make these open access particularly when it comes to the full text screening decisions and general recommendations for most of the evidence synthesis guidelines are that decisions should be made at full decisions made at full text should be presented somewhere so that they can be traceable and of course not forgetting the importance of objectivity in any evidence synthesis so this is following the key systematic review principle of trying to be as objective as possible and of course we know that there is a degree of subjectivity that creeps in but it's about being transparent about what you've done so that we can assess whether a review has been done to a different different levels of objectivity so during the screening process this is where we're really trying to minimize the risk of selection bias and also the differences in selection bias between different reviewers and this comes in by having multiple reviewers multiple rounds of consistency checking between reviewers and this is also known as inter-rater reliability as well and on that point where we've got multiple terms yeah I just wanted to put this up here because I find well I certainly found when I started doing systematic reviews systematic maps that there was quite a lot of terminology flying around and I wasn't necessarily sure what each of those meant and it turned out that quite a lot of them were meaning the same exactly the same thing so I just wanted to put this slide up here and kind of go through some of these and this is particularly important perhaps if you're in English isn't your your first language so the screening and relevance assessment this describes the stage that we're talking about the article screening stage sometimes called relevance assessment a reviewer and a screener that's essentially the same person if they're undertaking this process so you can use those you can see those terms used interchangeably and sometimes these terms will be used interchangeably across the method section of one particular paper if you're reading an inhibited synthesis paper so yeah it can't be quite confusing inclusion criteria exclusion criteria relevance criteria eligibility criteria these are all the same thing as well essentially although I would say that inclusion and exclusion criteria are set into two separate you might see them as columns in a table so you might have a list of study study examples or parts or facets of a study which are which would denote a relevant review study and then you might have a list of those which would denote a excluded study a study that's not relevant so that's where those two differ between those and relevance and eligibility criteria and then we've got interator reliability inter reviewer reliability or consistency checking or the level of agreement essentially this is how how consistent each screener is in looking at each of the studies so you might have a test set of say a hundred studies and you'll see which which review is on which studies was their difference in opinion as to whether that study should be included included or excluded and then we've got eligible and relevant these mean the same thing essentially a study that is to be included in the review scoping and testing I've put these as similar or the same things because yeah typically scoping is a testing phase of the review and we often talk about this when we're doing screening with your scoping stage to test out your eligibility criteria and also to train systematic reviewers as well and then in the last row we've got record article and studies and these these are referring to the the documents that you're looking at and your rating but it might also be that within one article there might be multiple studies there might be the case where you have one study which is reported by multiple articles so that's the difference between those two but then we've also got record and record is something that you might have come across if you're doing a lot of searching so this is your search hit whichever whatever that is that you get back from your search strategy you might have nine thousand hits and those each of those hits would be a record so I hope that's that's cleared up some some terminology just a quick pause to see if anyone's got any questions yet great so far there are no questions in the chat no questions on twitter and no questions on slack I'm not sure if the participants may want to open the mic and probably ask you something directly yeah so do feel free to open up your mics any questions we're just getting into the meat at this work now so let's move on to eligibility criteria so at this stage at the screening stage any of your articles the screening should be taken undertaken according to particular eligibility criteria and the aim really of having this set of criteria is that each article should be judged on the basis of a set or predefined set of criteria which should avoid selection bias as much as possible and this is one of the I would say this is one of the major criticisms of traditional reviews is selection bias it's where the authors have or have not sometimes just have not reported how they've decided to include studies in reviews so it's a really important stage to be transparent and repeatable at and for you to be able to make your decisions in an as objective way as possible so we need to treat all articles in the same way it should be repeatable and any articles which are included should be included on the basis of their relevance not on the significance of their results and this particularly comes in when you're reading full texts of papers sometimes it's a good tip to try and avoid reading the results section of any full texts until after inclusion because then we're removing the possibility for subconscious changes in our behaviour and sometimes this isn't possible when you're looking at abstracts and the articles report the results and the conclusions and the abstracts but that's certainly been something that's been talked about so we're assessing each potentially relevant record or article against this set of eligibility criteria and it also helps us to prevent changes in our inclusion decisions over time because sometimes we get what's called decision drift which is where anyone despite your best intentions interprets the eligibility criteria differently as time goes on now there may be points at which the eligibility criteria do need to be altered as time goes on and that sometimes that comes because only after a certain point do you realise that there's a set of studies or there's another type of outcome which is being reported by studies which wasn't necessarily picked up on when you were doing the scoping or the protocol development so hopefully that shouldn't happen because you should have done comprehensive scoping but if it does that change any changes should be recorded to eligibility criteria and how do we define them well we often usually base them on aspects of the review questions key elements so the typical pico question the population intervention comparator and outcome but also I would recommend that you would also base them on things like the study design perhaps the location the geographical location the perhaps the date there might be date limiters as well that you want to but also you do need to make sure that you justify any of these limitations why should it be a particular region or date and that will come through not within the eligibility criteria themselves but in the description in your report so let's have a look at an example of this eligibility criteria this is taken from a review which was undertaken several years ago now but I've tweaked it a little bit so this in this example the systematic review was trying to answer the question whether marine protected areas are where effective tools for sustainable fisheries management so we're really looking at the biodiversity impact of marine protected areas so areas of marine reserves sometimes and this was actually in temperate zones so in in those areas so the study eligibility criteria we have listed the key points about those criteria here so the population of any study should have included or could include any of the marine biota the intervention was a marine protected area in a temperate area now you might actually say that this could the temperate part could actually fall into the population as well or it could just sit on its owner's geographical location that doesn't really matter the comparator they wanted to include less protected or unprotected areas to compare the protected areas with and then the outcome was the a measure of the biodiversity impact so that's here density and abundance measures biomass species richness and also they've also got study design here as well so each study should have been an primary study collecting empirical research um now this is another point um just an interesting point actually because I think there are some differences in how how people perceive what empirical research um is so typically for me it's something where the the um the authors of the study the researchers went and deliberately collected research for their own purposes um but it might also for some people this means a a re-evaluation or a reinterpretation of existing evidence so I've seen this happen in different fields where there's a bit of a difference in interpretation even on things like a study design which I thought were quite um were quite well recognized as the same thing so it's definitely important to make sure that there's consistency and understanding across the whole of your research team so how this is one of the questions that often comes up how detailed should you go with your editability criteria um now the example that we had back here is actually not very detailed at all um it's really basic um and I just wanted to show you that because um we've got an exercise using that in a few slides time um and to give you an idea of how how it is to undertake assessments when you don't necessarily really know um the details of the editability criteria and a short answer to this question I would say as detailed as possible um and sometimes this detail might need to need to be added in as you go along the review so as an example I have um taken a screenshot this morning um because I didn't know whether this was actually quite um difficult to understand how detailed you should go okay so these are six pages of um inclusion criteria for one systematic map that I was undertaking um so you can see that this one's the six pages and it's got a number of notes um and points and different categorizations within it um so you might think this might be overkill but it was really helpful for us and the whole review team when we were doing our screening and I've just taken a screenshot of the first the first page here of that eligibility criteria um so you can see that in terms of the population we were looking at individuals, households, communities living or working in areas uh closed deliveries in Southeast Asia so we had to define what countries were in Southeast Asia so we literally had to go through and decide well there were several different definitions of what Southeast Asia is so economic definition the geographical definition uh we also needed to define what coastal area was so how do you define that some people might see it as you know 100 or 200 meters from the beach maybe that's what an ecologist might uh might determine a certain coastal zone as but actually we were we wanted to be a bit broader than that um so we defined them as areas which are adjacent to and heavily dependent on or impacted by the sea so that could be in economic, sociocultural or ecological terms we also said that each of the studies must clearly state that they're focused on that relevant population so you can start to see here how um how detailed we're getting with these eligibility criteria um and then when it came to the intervention we had a number of different categories or subcategories um so just for those of you who are looking and have more time to look at this um offline this intervention was about um marine management interventions so we had to try and classify those into different types of marine management because again it's something that could be interpreted by different people in different ways so I don't want to scare you but um it's it's more a point to just to say please be as detailed as possible and you're probably not going to realize the level of detail you need to go into when it comes to do this so this is a point where scoping and testing is really important so in terms of how the process works it generally gets undertaken at several different stages so these can be um title abstract and full text we've got those sort of three levels of information that any one study might give us hopefully um and this helps us save time by consolidating the similar activities um and also by screening out at title stage you sometimes are able to remove a large number of irrelevant titles if you are not able to do that by um automated text mining um methods um typically though I would say a lot of the um screening gets undertaken at title and abstract and this again this will sometimes depend on what your tool or your software is that you're using and how it presents um each of the records so if it presents the title and abstract together it's a lot easier to just screen titles and abstracts so just screen the title first and then if you can't decide make a decision from the title then you screen at the abstract level um so often you'll have multiple hopefully you'll have multiple reviewers doing screening that's the most um sort of objective way and the most reliable way of doing screening um you should be recording your reasons for exclusion now I would say people can do this at title and abstract stage but if you've got anything more than a few hundred results then you you're really going to be struggling to do this and um manage your resources at the same time so typically just recording the reasons for exclusion at the full text level is sufficient and it's typically included as an appendix and supplementary information of any published and evidence synthesis so here's another one uh what happens if I can't make a decision so this is very typical um we often get people doing title screening and title and abstract screening and exercises and there's just maybes across the whole lot well maybe or I can't tell yet um so if you're screening at title and or abstract stage um if you have any doubt about inclusion you should be including that review it might if it's potentially relevant even if the information is not provided that's the moment at which you should be inclusive and that's to reduce the risk of eliminating any um non-relevant studies um yeah what happens if I can't make a decision at full text level um if there's not enough information then you may need to go back and contact authors if possible again it's all about being transplanted at this stage okay so we're going to have a go at screening um some titles against some eligibility criteria uh we'll use an example from again this was the a real systematic review but it's kind of slightly condensed a little bit um and that review question how is the level of protection in a marine protected area affect populations of marine living organisms okay so I'm now going to ask those of you who are in the meeting today um to speak up because I want some answers to this question I'm going to ask you when you identify the question components here so the question components this is a keychain question population intervention comparator and outcome so would anybody like to tell me what the population is here it's all the marine living organisms basically yeah absolutely right thanks valentina all the marine living organisms marine biotech as we said before the intervention then protection that's right yeah thanks depra it's the level of protection um in the marine protected areas so what would the comparator be here well the level of protection absolutely yeah thanks valentina in this one it's um it's one of those reviews where there'll be different levels of protection so you'll have won't just be uh no protection it could be different levels and finally what's the outcome can be density can be because it affects population so density live or death or I don't know something like this yeah absolutely right valentina um affecting populations so when we say populations we mean the sort of absolute numbers density and things like that so yeah a little bit easier that one because we've already seen that one before but thank you so much everyone for uh helping um and now let's move on to the exercise I'm going to so here we've got the inclusion criteria and I'm going to put up five or six I think different studies okay um and these are just the titles so what I want you to do for the next few minutes um we'll give you let's say four minutes here um to look at each of these and decide whether you would include it or exclude it based on the title alone so off you go okay um I'm gonna open it up to you guys now so who would like to tell me what your answer was or number one is it include or exclude exclude exclude thanks Peter yeah so why did you exclude that one because Nepal doesn't have a coastline excellent thank you so that's uh number one it's the wrong population for that one so and if you were doing this at full text you would write down no population you could do if you wanted to as a reason at this stage but for title stage it's probably a little bit too much information uh for the next one number two same wrong population but in this case because it's not um marine biota biota is poverty so it's probably human population yeah okay so you've so Valentina you've excluded on the basis that um it's on human populations which is which is a fair assessment now I don't know anyone else has uh have a different opinion on that one yeah I would say maybe but you'd have to go to the full text to see if they've included um outcomes that are of interest beside poverty yeah I think that's what I'd probably say as well Peter that's a really good assessment so although it's only mentioned for reducing poverty here it may have also I mentioned some of the biotic measurements so I would say you're right but the only other thing is that we are looking at temperate marine protected areas and these ones mentioned Costa Rica and Thailand which are tropical so probably on that I would exclude so it is quite useful to actually know what what reasons we're excluding on so thanks both of you excellent um for the next one number three I can go again I see no reason to reject this some marine reserves doesn't say anything about where it is um yeah except yeah I would accept that one as well definitely that looks like a pretty spot on from the title only so we'll take that through to abstract and full text the next one number four well for this one I would probably go to read the abstract because I don't actually understand what is about from the just from the title yeah I work in I work in Africa so I know Zambia is landlocked so I'd reject it on population great absolutely it's a bit like the okay cool yeah thanks thanks Peter um yeah we're probably not the only one you've got that um a little bit wrong there um Valentina um when looking at these sometimes we do need to have a map geographical map up just to make sure that we will um we know what we're doing so yeah excluding from the wrong population but otherwise I would say that's a very um it's a very good point um again not really not really being sure as to whether um whether they've got the right uh outcomes so number five I've worked with this person so I know what this paper is about but uh but um yes you reject this it's not about um marine ecosystems at all it's about terrestrial organic farming systems yeah I would say yeah for me it was rejected too because I mean organic farming and well I don't think that it is the right population yeah it's not it's not a protected area um so yeah excluding on those basises and that's a really good point actually Peter that you bring up um sometimes we do come across studies which are written by colleagues people that we know or even ourselves sometimes if that does happen we tend to say okay um let's uh for those ones we need to get somebody else to do the assessment on it because we don't really tend to review our own work in terms of eligibility and finally number six included I don't see any reason why to exclude this one marine reserve canary island perfect yeah it looks good it looks good on the face of things one thing I would say is about again it's about the temperateness of the canary islands so they are they're a Mediterranean climate um but in this case we're not sure whether we're only told that we have to include temperate marine protected areas so in some in some classifications people group just temperate um and tropical um boreal into like really big groups of geographical zones so if those are the zones that we're talking about then yeah I would say that this is probably in the more temperate zone but if you're classifying them in a much more detailed ways perhaps using the the Copenhagen classification system um these are going to sit separately to I mean the temperate ones and I've done a paper like this before and had to define exactly what the feed the kind of Copenhagen temperate zones were and I needed to make sure that each of I had a map with me every time I was doing my screening so I needed to make sure that the particular location of that study was placed in um the right zone so well I hope you found that exercise interesting a little bit fun I guess um we are now going to do it another time so I'm going to give it another go on a slightly different topic though so here's a systematic review question what are the human well-being impacts of terrestrial protected areas um and let's um let's go forward to the actual exercise how I'll give you a little bit less time maybe a couple of minutes um so yeah take it away here it is about population is humans living in or around a protected area your intervention is that protected area establishment or just presence of it the comparator is a less protected or unprotected area um and your outcome is measures of human well-being and again we're including any empirical research Jackie there is a question on slack why would you do the title and abstract screening separate is it not more efficient to screen it in the same time yeah a really good question um it again it depends on how you're you're um seeing your okay so it depends on two things so first of all how it depends on what the tool is that you're using so some tools um don't allow you to see the title and abstract together most of them do but some of them it's easier just to see the title um and so in that way it can be quicker just to screen screen the titles first um and also particularly when combined with the fact that um in some reviews and because of the way that some searching is undertaken or can only be undertaken for some topics you might know that you're going to get quite a lot of irrelevant records that you can't you're unable to screen out now maybe text mining is going to help us with that a little bit more in the future but if that does seem to be the case that you're looking through and you're actually you're bringing up the abstracts but it's not not helpful I would say then just go to just screening the titles first typically I would tend to do titles and abstracts together because that's the the softwares that I use show them both and I would say it is more efficient to do title abstract together generally I'd say also just try out and see how you get on which which method you prefer for some reviews I have gone and done titles first I quickly looked through the titles in fact there's one review I'm doing at the moment which I'm doing that just give it a few more seconds so people can finish these Jackie so far there are no further questions here on the platforms also I've not seen something on the chat if the participants maybe want to ask something that would be great yeah do feel free to to speak out at any point um let's move forward so who would like to tell me what they gave for number seven um I can go I would include this one you want a really difficult strain product error management seems good yep it's about humans it's about protected areas that's all we can tell for the title so yeah let's include it absolutely number eight I'm pretty sure it's not just Valentino and I understand I'm cold so everybody else who's trying to pitch in but I would include this yeah I did the same include that one not quite sure exactly what it's about but it certainly got relevant elements in there great number nine I would say no so this one because international trading status um is um species protected species but not product in the area so I would say no yeah it doesn't look it doesn't look relevant from first sight however I would probably say I would just include this one because it might be talking about talking about the role of Thailand it might be talking about some protected areas within Thailand or maybe Thailand's um a designation of areas reserves which might be impacting societies uh reptiles and amphibians there might be human well-being aspects in in that as well if it's it's talking about how people are exploiting animals so I would say it's very unlikely to be relevant but if you wanted to be really inclusive you could include it at this stage but I generally agree with you Valentina number 10 somebody else okay looks like it's me again reject this this looks like it's some kind of um paper about the genetics of wild canid species I don't think it's about doesn't seem to have any links to humans or human well-being or even well the reasoning to rather might be a protected area but I think it fails on population and outcome probably yeah I would say the same um we've also got the next one which is number 11 socio-economic impact of the karu national part now I would say that's probably a probably an include we've got national part we've got a protected area we've got socio-economics it's got to be talking about humans so I'd say that was exclude and then the final one number 12 I think we could probably all say that one is going to be a definite exclude that's not in our inclusion criteria so let's move on um I've also got another screening exercise which is looking at the abstracts um but I'm actually going to skip over that right now um and just kind of move forward and talk about um in fact no we'll go we'll go there we'll go there um so if you want to go to this link um I think the the piano might be able to pop it into the chat as well um it's an osf site and it should take you straight to a page where there is a pdf document um which says as mark on 2023 workshops screening if you click on that um it should pop up or it should pop up automatically hopefully so do let me know if you've got any problems accessing that um but again I just want to give you about the same amount of time you've got four abstracts in that document um and if you want to look at that document and trying to assess it's on the basis of the same exclusion inclusion criteria from before so it's about human well-being and protected areas um yeah so just go to that one and I will just put up the the next slide we'll also have the same information so you've got the website there you've got your inclusion criteria and you've got those four studies so those are the titles in fact there are some of the ones that you've already looked at just in the last few minutes um I'm going to give you just a few more minutes again about the same amount of time because you don't actually get you can't really take your time about this you've kind of got to to move forward if you've got a lot of um studies to get through so it's really working out that trade off between how much time you're spending on it and how accurate you're being and of course you're going to get way more efficient as you go along um so I'm just going to give you a couple of minutes to look at this now thanks Bibiana for posting the link I'm not sure if you're able to do that on YouTube as well certainly be there um after the live stream I've also shared it on Slack so I think that we can be reaching more people okay so that wasn't very long but um hopefully you've got a good feel for some of those studies um pretty sure you'll get very used to um skin reading things I certainly do I think even more so since I've been working in systematic reviews right so who would like to tell me what you thought of the abstract for Anita tell include or exclude or not sure I think in general I find it um difficult to know if there's comparators in these um from these abstracts so for Anita I thought it really doesn't sound like there's a comparator it sounds like they're just focusing on describing a single protected area um now I don't know if you want to be conservative and say but you need to check the full text to really understand if that comparator is there or not um so that's what kind of I thought yeah it's about humans that's a protected area and this issue's around human yeah I think human well-being probably um and it seems to be empirical research so I think that's all fine it's just a comparison for that one I apologize it seems that Jackie has some technical issues okay now she's back Jackie you are muted hi everyone sorry about that I just dropped out with my internet it's it's raining here in the UK um for a change so I'm sorry about that um I managed to get to this stage where um we decided that Peter had excluded the uh Anita study sorry included the Anita study and I would say I would also agree with you Peter to um include if unsure um whether you're whether there is a comparator or not okay let's move on to the next one Barney Jackie probably you could share your slides again we are not seeing the abstract thank you so who uh so hopefully you're seeing my screen now we're back online um so who thought Barney was an include I thought it was similar to the previous paper and that they've looked at protected areas and their benefits to people it's just unclear if there's a comparator in their analysis or not yeah I would say the same unclear if there's a comparator does mention under managed protected areas so maybe that could be a comparator not sure if that would fall into our um inclusion criteria but we'd have to have a look thank you Peter uh the next one uh niche man well this is the one which I said let's be um let's be a bit inclusive about this and it turned out as I was almost certain that it wasn't relevant it's not about protected areas um I don't think there will be anything more about that in the full text so I think we can exclude a bit the stage and then the R school by Simon now this one looks this one does look really relevant um it's about a national parks we're assuming that it's protected sometimes some these national parks can just be paper parks but um we're assuming that it's protected again I'm not entirely sure of this comparator so this is it's a good point you've picked up Peter it's typically the the comparator which isn't shown through um the uh in the abstract or the title so yeah definitely if it's not there I would I would include it and just to look at it at full text just in case because we don't want to be overly exclusive so I mean any reflections on that process so far Jackie there is a question Jackie there is one question by Ismaila Salaw not about the exercise precisely but the question is sometimes in Africa we do not have access to some journals because of subscription fees what do we do we include a study we can't have access to what implications does it have for publication bias a really good question um I've actually got a slide about that later on but we'll we'll cover it now um well it's not a slide it's just a prompt for me to answer that question it can be really difficult to get access to a lot of papers um there are a number of methods that can be used um so um you can you can always ask the authors if you can get hold of the authors from the email address you can ask them for that um there are a lot of a lot of work is being published open access these days you will actually find that's that's quite um encouraging and preprints can also be um uh like draft versions can also be made available by by authors too um there's becoming a bit of a backlash against the the the paywall so there's another a number of ways in which people can help as well there's there's a hashtag a twitter hashtag called han has pdf so i think it's hashtag c a n h a z pdf i think that's work um and that can sometimes help identify you by trying to find the pdf of a study um but it is a problem if you've got collaborators working in in different organizations on the same project that can be also really helpful um because you can ask them to to share some of the the work that they're doing if you're working on the same project so that's why international collaboration can be can be really useful and really important um your library might also have a little interlibrary loans section um that can be expensive but um it's one way of obtaining difficult defined um studies as well so i think one of my main reflections on this is that there's a huge variability in what people um determine is relevant and not relevant particularly when your inclusion criteria aren't very well defined so that makes a really good case for trying to define them a bit further but sometimes you just don't know and you have to go to the next level to find out so some key things to remember um if you're uncertain retain the record um it's easier to apply if your inclusion criteria if they're well defined so as i mentioned some of mine have been defined to the nth degree um the other important thing is to avoid vague terminology or at least if you're using terminology to explain exactly what you mean by that so we said by marine management interventions we explicitly um listed and we had some examples of the types of um of marine management interventions we were interested in you could be trialling your eligibility criteria um and that trial should involve everybody who's going to be carrying on carrying on doing that systematic review and screening process and again recording and reporting and the final review is really important um here's a question that also comes up quite a lot so what about how many reviewers do i need to include in my in my evidence synthesis do i actually need multiple reviewers i've only got myself sometimes um if you're working on your own project phd students sometimes don't have access to a lot of people to help them um it can be a problem so one thing that you can do if you're you're struggling for people is to um to sort of work share so there's um you can do this informally with um your colleagues um where you help them out on one with you and you might um and then they can help them out on another one help you out on another one um there's also a there's a task share um kind of forum i suppose or group which you can access there's one which runs through the Cochrane collaboration so i'm not sure if that's relevant for for many people but if you want to go and look for Cochrane collaboration and task share um there's number of other ways you can you can share work in that way um how many of you is doing actually need well i would say minimum of two if you're going to be doing um double screening or uh consistency checking so it's also very useful to have a wide um breadth of experience in your review team when you're doing screening so topic expert is obviously really useful but sometimes a methodology expert um can be useful as well perhaps your topic is spreading across many different themes and in that case you might need somebody for each of the themes um so it really depends on the on the type of project lucky there is a question on slack peter stewart asks any wisdom on free tools for automating the title abstract screening process oh that's a really good question um i will go through tools at the very end um um but i would say automating title abstract selection is a very dangerous process to do if you're going to rely on it completely i would say there is there are definitely some ways in which the tools have been uh developed with this in mind um and so i wouldn't necessarily try and develop or use a tool which completely screens out um all of the the studies that title and abstract level because i would want to go back and check um there's a few which do uh a ranking which i find quite useful um one of those is sis rev and i'll i'll talk about in the next um next uh at the end of the session and there's also um yeah there's a couple of others but i'll get into those later um right okay so we know that we're going to get differences between reviewers so we do need to assess the level of agreement um and you can do this using a consistency test and then if there is this agreement you can refine your definitions and repeat the assessment so you can do this with with a consistency like a test which i'll show you one in the next few slides um but probably one of the best ways to do this is literally to sit down with people in the same room or to get together after a certain number of screening records double screening or triple screening records and um having a discussion about the reasons for differences um and that process should of course be recorded and tested and scoped um put a slide in here for any questions um in case any wins not uh not quite a holding on at the moment um but i want to just cover quickly uh consistency checks so these are really important so you're going to have multiple reviews in your study um and even if you're not having multiple reviewers and you don't have the time that's not recommended but if that's all you can do you should also be consistency checking across the time period that you are um screening and that's to minimize the risk of this decision drift so um it can be also be really fun by doing consistency checks i did a recent review once where um prizes for those who were getting through the screenings and having the lowest consistency um problems with consistency um keeping notes regularly uh discussing we tend to do a 10 percent sample as a very minimum but this um is i would say it's a minimum and i would repeat this across the the process perhaps um two or three times it depends on the scale and the size of your review of course so one of the tests for consistency which is often talked about is the cohen's kappa test um i'm going to flip through this quickly because uh i want to get on and talk about um things like tools and give a chance for people to discuss this um so the cohen's kappa is a measure of this inter-rator or inter-reviewer agreement level um it basically in a nutshell it tries to assess the proportion of agreement after an element of chance has been removed um this is the calculation for it you can find plenty of information about it online and some tools that will help you calculate it as well um so typically this is what you would have if you've got say two reviewers and you can do this um what's a slightly different version of this test with um more than two reviewers so you've got the includes and excludes by both reviewers so where both reviewers agree to include both agree to exclude when agrees to include and exclude and vice versa and at the bottom here you can see there's a um sort of like a table of um interpretation of the kappa value so if your kappa value is in this case it's falling at 0.5649 it falls within the moderate agreement level um now sometimes cohen's kappa can be a little bit misleading um and this is particularly the case when um you have rare categories where you have um you're mostly agreeing on some studies then there were some which were um only a few say like five or ten percent which you disagreed on and in these cases the kappa can underestimate the agreement on that rare category um and this is due to the way that the statistic is calculated um the difference in um quantity and allocation between these different uh boxes that we just saw in the last slide it's typically and sometimes can be considered a overly conservative measure of agreement other ways of measuring this are percentage agreement um on particular how many you agree to include or exclude and there are a number of other alternatives as well there's scott's pie which i haven't used fleece's kappa this is a to be used when you've got multiple reviewers not just two um but you can also use weighted kappas as well what i would say is any of these tools are just indications of agreement levels and you can use them i'd certainly say it's better to use them than not use them but also use it alongside um your own um sort of sensible judgment typically if you are repeating your consistency checks across the the course of your project um you'll be able to get a good feel for how much decision drift there is or how much difference there is between reviewers and you can adjust and talk and um provide more detail on the inclusion criteria according to that jackey there is a question on slack at which stage would you test for inter-rater reliability i can imagine results will be different in title screening compared to full text screening yeah absolutely so let's take title abstract um screening um together because that's typically what's done well you would do them at both stages so you'll do it both at the title and abstract stage and the full text stage that's really important um so yeah you've got to do it at any stage that you're doing um screening um so you might be really confused now what am i going to do i don't know how many reviewers i've got if i do have lots of reviewers i don't know how i'm going to to record all of this or or to overcome this potential selection bias um well just remember that everything you're doing in one of these evidence analyses if you're recording it you're already doing a lot better than any other review type so it's all about transparency i would say just keep rechecking all the way throughout keep taking a random sample that's another really important point is that when you're doing your sampling so say you're going to sample 10 percent of your studies at the very beginning um you need to do that randomly to if you're doing a consistency check between different people because um if not if you're doing it by author for example you're just taking the first chunk that appear in first 10 percent that appear in your list of records um you might be getting a load of them that are undertaken by a few authors um and that can that's not a random sample of your of your full data set um so just running through um the the next part is about how to document what you're doing here so you might be familiar with flowcharts here's one example um and we've also got the example of the roses in the next one just to mention here this is also where articles and studies um delineate so where you might have multiple articles in one study or multiple studies in one article and that's a really important definition to make which will fall out when you do the sort of later bit of screening and data coding and data extraction this is the roses um flowcharts set up so many different ones that you can use this is one which was developed for um in collaboration for environmental evidence mere synthesis um and just a note about full text screening it will need to follow the same principles as the title and title and abstract screening um so that's in terms of doing the consistency checking making sure you're using the inclusion criteria includes an exclusion criteria and fully um but also to mention that sometimes it's you can find that actually when you're doing the screening at full text you're reading the full article this might be the most efficient point at which to um do your coding so that's where you're taking um variables information out of your um each of the studies and recording it perhaps in an excel sheet or something um so just wanted to make sure that people uh are aware that this can be a really efficient way of doing your coding doing it at the same time as screening so we've had this question already about not being able to find find or access a full text um research gate can be another really useful um source of studies um sometimes i would find find that actually i've had a really good response rate from authors by asking them really nicely in emails so now that we typically get email addresses with the publications of papers uh it's a it's an opportunity for you to request information from from authors and this also leads into the kind of the coding and data extraction i've often found typically that there's not enough information in a lot of the studies i'm looking at um and i've sent well structured but um copied and pasted with copied and pasted parts uh emails to authors and i've had a really good response rate so if you're you're asking nicely if you're saying what you're doing it for and also um but then if they're going to help you they would get a citation in um of this of their study that can really help you um so other common pitfalls here um these are quite practical ones so not taking regular breaks when screening so this can lead to decision fatigue where you kind of get a little bit sloppy in terms of your decision making when you're screening it can also really burn you out and make you demotivated on the project particularly when you've got a lot to do um and it also leads to decision drift so um that's one of the common pitfalls do make sure that you kind of struck to your your screening and it's good to do it in chunks but not too big chunks so you'll find out what your own tolerance levels are trying to code in too much detail so this might be when you're just making decisions and so for example at the full text level try not to put too much information in there because it's only going to slow you down and do make sure you get some as well um this is something that often happens and it means that you might need to go back and re-screen some of your studies it's when you miss potentially relevant or potentially interesting studies um and I say potentially or sideline because um these studies might not have been anticipated when you first started doing your review and it might be that you've suddenly realized you've got a whole chunk of studies I've got this happening in a review that I'm doing at the moment I've got a whole chunk of studies that I'm interested in but I don't think they're quite relevant what I've done is I've tagged them so I've said um these studies are included but I've got another column in my excel sheet where I'm saying okay we're tagging these because they are on a particular topic or they are perhaps they're undertaken in a geographical location which isn't relevant to your review but there's a lot of them you might want to include them as a separate group to come back to if you have enough time or if you've not got enough studies in your full review and again that's perfectly fine it's just a case of recording that what else did I want to say about that don't remember uh so yeah not keeping track of your consistency or not keeping track of anything um is a really common pitfall and I found that out to my own detriment so so do try and make things um rather than repetitive and tiring more fun and competitive um again as I said I've done this where I've given um given prizes to people for doing various different screening um minimizing amount of detail to record and tag interesting articles those are some key things so in summary uh about this part of the screening before we move on to the tools we've seen uh that we should be using these really well-defined eligibility criteria for efficacy efficiency transparency and repeatability and we know that there's going to be a degree of um decision difference between reviewers your aim is to reduce this and for any situation in which there might be um difference in decision that the authors should be more inclusive than exclusive and that will account for that difference in decisions we've talked about assessing for repeatability and checking um criteria documenting everything using flow charts um and keeping a detailed record of any record any reasons for exclusion so I just want to finish off by talking about softwares now um so there's lots of different softwares to manage articles I don't know if anyone's looked at the systematic review toolbox um aimed at systematic reviews but relevant to many public instances and you can go on this and set tools that you want to have the stage of review that you want to look at if you look at screening there are tens of um tools that are available so let's start off with Excel and other spreadsheets they can be quite easy to use your many people are already familiar with them so if you've got a multiple reviewer review team then uh it can be um you know it's quite easy because they already know how to use it they can be um they can be clunky though um they're not designed specifically for doing evidence synthesis but generally they're inexpensive they don't cost anything you can also use reference management software so thing like EndNote, Zotero, Mendeley um these can be really useful and I know some of these the searching group like to use uh reference management softwares when it comes to screening as well these also can be quite easy to use once you're used to the system but they can be expensive and it might be that different reviewers on your team prefer to use different reference management software so some are really anti some people who use EndNote are really anti Mendeley for example um so that can cause a bit of friction but then there are review management software so we've got things like Covidence, EpiReviewer, Kadeema, Cessrev, Ryan these are all specifically designed for the purpose of undertaking reviews um however they can be expensive um and that's a very brief overview of what those ones are let's look a little bit more at a couple of these um so I would say that EpiReviewer is one of the ones which has quite a good bit of functionality um it was designed by a group of researchers um in the UK and it does a lot of different things they one of the good things about it is they have a fantastic support and help desk um they are very quick in getting back to you um with any problems you can always access your review so even though you might not be subscribed you can still get back to it um and um it's relatively relatively inexpensive so that's one worth checking out there's also a completely free one this is called Kadeema this was developed by the wider Collaboration for Environmental Evidence Network that's one of the networks who produce guidelines for systematic reviews and environmental uh syntheses um it's also based on the EpiReviewer model but it's slightly tweaked and designed a bit differently that's also one worth checking out particularly for those who um want access to the full kind of uh the full software without having to go just to a limited free version so I would also recommend having a look on systematic review toolbooks I've picked out a few of the ones that I've used personally before here um all of them have got their pros and cons and I'm unable to kind of go through those um with you right now um because there's quite a lot of them but I would say that one of the some of the recent ones I've used are um well cysred I've used is the most recent one that I've used uh it is freely available I don't know if you have to pay for it I don't know if there is a paid version I'm not paid for it if that is the case but it's got good um functionality there's you can use it on your phone it's quite easy to use on your phone um I think there's an app for at least a couple of these I can't remember which cysred's got an app but Ryan does um so it's been quite useful when I've been actually sitting on a train or something I've been able to just go through and use use these tools and screen um quite easily um Ryan has been useful there's a bit of a quirk in the way that it exports some of its um um data I'm not sure if that's been sorted out I know they've had a recent update and revamped so there may have been some changes there um and a lot of these have free trial so co-edents for example that is quite an expensive one but I've used it with um on some commercial reviews that I've done and it's been pretty pretty useful actually um but for example this one for co-edents for a single review it's 240 US dollars per year um Ryan for example one of the cheaper options to go for is four US dollars per month for students um and for professionals it's about eight US dollars per month so there's big variability in the in the price um that's available um what do I want to say other about these but yeah so a lot of them are using now are using either machine learning um or text mining I think mostly machine learning so for example with cysred it uses and I think with Ryan as well it has a ranking system so it uses machine learning to determine based on these studies the titles and abstracts that you're looking at which are the most uh which are most likely to be more relevant and then it pushes them up in the list of review of studies to be assessed so that I find is quite useful um cool and as I've mentioned this is these are becoming more and more available um we've seen that actually by doing using machine learning or building it in two um tools for systematic reviews the cost of there was one study which worked out that the cost of a review could be potentially halved or even less than halved by um using single screening with text mining as compared to double screening so there's a lot of opportunity there and just to also shout out to Martin Westgate GitHub rev tools um this is a package for article screening I do believe it uses some uh text mining or text analysis and there are plenty more talks on the more technical aspects of um text mining used for screening and text mining itself on the Esmalconf website if you want to go and have a look at those and some of them have been presented in previous years so I just that all that remains for me is to say thank you and to to field any questions before we need to sign off so do feel free to ask any questions Jackie there are so far no questions on Slack, Twitter, YouTube, here in the chat okay so there is one question on Slack now one eligibility criterion is language not all studies are published in English in my experience translating documents from German, French, Russian, Spanish, Korean or Chinese to English using Google translate works fine in principle would you recommend collecting and screening non-English documents as well as English documents although this certainly depends on the purpose of one's research yeah I think you've hit the nail on the head there it really does depend on on the research that you're undertaking that topic area if you think that it's likely that there's going to be a large suite of studies written in a particular language I would say yes go for it um but otherwise we typically tend to only search in English and only um only only look for studies in another language if there's a good reason for it and that's perfectly justifiable um in in a in the right of review okay so I think that we are all set and the questions have been answered successfully yes great thanks so much okay thanks thank you so you Jackie so that's it for today's workshop on screening studies for eligibility in evidence synthesis we hoped you enjoyed it as much as we did thanks so much Jackie for this wonderful workshop we learned a lot and see you all soon at the next session