 Good afternoon everyone, or good day to everyone, wherever you are. Welcome to ESMComp for 2023, and this is workshop 10. How the feet framework can help you select study appraisal tools suitable for your systematic review. This workshop is being live streamed to YouTube and has a group of participants taking part live and really well warm welcome to all of you. Do you have any questions for our presenters? You can ask them via the ESHACAFON Twitter account by commenting on the tweet about this workshop. If you're registered with the workshop you can also ask a question in the Q&A in Zoom and also you can comment and chat with the other participants on our dedicated Slack channel which was sent along with your registration information. We'll try and answer all the questions as soon as we can. In addition we'd like to just draw your attention to our code of conduct which is available on the ESMComp website at www.esmicomp.org so go and have a look at that. With no further due I'm going to pass you over to Jeff and Paul to take you through this workshop. Thank you guys. Welcome. Thanks everybody. Welcome along to this workshop. It's great to have you here. I do apologise we had a few technical hitches. I guess no surprise and that's why we're just a little bit late starting. Fingers crossed we won't have technical hitches for the remainder of this workshop. One of the things I'd like to have done is to invite you to say where you've come and joined us from around the world which would be quite interesting. We weren't able to put a poll in for that but if you'd like to introduce yourselves in the chat that would be great. So we have an idea about where you've come from, which organisations you work for. So today we're going to be talking about the feet principles. Now you may well have never heard of these. There's something that's relatively new principle so you may well have not heard of them. And I'm going to talk about how these principles can help you to select study appraisal tools when you're conducting a systematic review. And so we'll tell you what the feet principles are and we'll go through a worked example of a critical appraisal tool in applying these principles. So before we do that just a bit of an introduction. So Paul would you just like to briefly introduce yourself? Yeah thanks Geoff. So I've been watching Geoff for quite a long time. I think we first met at CE 2016, 2015, something like that. So I'm a research methodologist. I work in the environmental health space so understanding the impact of the environment in which people live on their health. I do research methods in that space. I work for open science practices and I do quite a lot of work around publishing standards particularly in my role as a editor-in-chief of the new Open Science Journal having the space toxicology and then my experience in systematic review comes from being systematic reviews editor at Zendron's International for seven years which I was doing till autumn last year. We're handled by 450 submission types and I can say confidently that critical appraisal is one of the most challenging areas that my submitting authors would have to deal with. So I think this this feet framework that really Geoff initiated we're hoping will be really really useful for helping people who are doing systematic reviews kind of overcome some of these issues. Thanks Paul. So just to say who I am I work at Southampton University here in the UK in an academic unit called South Hampton Health Technology Assessment Center and we have expertise in critically appraising company drug submissions for the health regulator here which is called the National Institute for Health and Care Excellence. So we do a lot of critical appraisal of evidence but prior to joining this particular academic group I have a long ish sort of history based on my PhD many years ago which was in the area of ecotoxicology looking at the impact of farming systems on the environment particularly regarding pesticides. So I've got quite a lot of environmental experience in my background and so I've kind of fused my primary research in the environment with my more recent health technology evidence synthesis research together and I'm actively involved with the collaboration for environmental evidence which is developing guidelines and standards in the area of environmental evidence. So that's why I have an interest in this area and I would agree with Paul that critical appraisal is quite a complex and challenging part of the process of evidence appraisal so hopefully we'll make a bit of an improvement on that this afternoon if it all goes well. So the first question we'd like to ask is have you actually done any critical appraisal and there's a poll link in the chat thread but if you've got a smartphone or a tablet and you can read a QR code fairly quickly you can have a quick point at the screen and we'd love to hear whether you've actually had any experience with critical appraisal and hopefully the answer will be yes otherwise we have to explain the very basics which could take months but it would be interesting to see. So have a go and then we'll pitch this workshop at the level of experience that's appropriate. So I think this is looking pretty encouraging Geoff. Do we have an idea of how many people have voted in the poll because my screen oh five people yeah I think a few more people could yeah that's it we're going up now. Be honest and be very honest if you're not sure if you've done some kind of a scientific assessment involving appraising evidence and you're not really sure whether it's critical appraisal that's absolutely fine to admit it's just useful to know what people's experience is. Yeah if you want to type anything to kind of expand on the answer you gave I'm sitting here observing the chat while Geoff presents and I will jump in with questions and their season things as they arise so just just don't be shy of using the chat box. I think the popcorn stopped popping Geoff. Okay so the good news is that most of us are familiar with having done some sort of critical appraisal so that's great. Hopefully those of you who haven't will understand and appreciate what critical appraisal is and what the challenges are but if you get stuck anywhere along the lines pop something in the chat thread and we'll try and help you out okay but for those of you who have done critical appraisal one word please on how we would describe it what's your experience one word we're painful in the chat painful is that somebody said something painful as in the chat no no uh yeah yeah I can relate to those words so far so yeah yes that's that's the word I would use I think be honest I wonder if frustrating is going to get up nobody's put the word easy nobody's put the word very straightforward nobody's put the word quick yeah difficult so Jeff we've got um about 12 live participants so I think we're probably yeah yeah that just basically reinforces what I think Paul was saying is you know we our experience is that it's not easy is it critical appraisal and so um and there are various reasons why it's not easy and we could spend the rest of the year probably discussing what those are but obviously we've only got a short time this afternoon so we'll focus on some specific reasons that critical appraisal is challenging and try and help with those so why are we developing a framework for assessing appraisal tools well obviously critical appraisal is a crucial part of a systematic review it's absolutely essential that we know that any of the studies in the review are biased so that we can take that into account in our data synthesis so we need to have some kind of a critical appraisal step in a systematic view so that's pretty straightforward the problem is that many published systematic views don't have a critical appraisal stage at all or they don't they don't have an appraisal stage that's very clear so we we don't necessarily trust their conclusions and there's a set of references at the end of this slideshow giving you some information about some of the problems in in systematic reviews that been published recently is regarding to critical appraisal so I won't go into the great detail here um but one of the key problems that we find with critical appraisal is how do we choose which critical appraisal tools to use um which I'll be coming on to but Paul um your experience in this area um is the environmental health systematic views are not particularly strong is that right yeah so we've done two bits of research on this we looked at a set of 75 environmental health systematic reviews that were published in the first half of 2020 yes 2020 and we were looking at how well they'd been done looking for kind of 11 basic methodological features that should be included in any systematic review one of these was a critical appraisal uh step and we found that while critical appraisal was quite common to be done so it was done like I can't remember top I have like an 80% of the systematic reviews we looked at only 11 of the 75 systematic reviews that we had in our sample had used a an appraisal process that seemed to be at least superficially valid so definitely difficult for people to choose and apply tools and then if you just progress one more so the other thing we've got is a while at environment international we use um an online appraisal tool uh so we can record the reasons why we're desk rejecting systematic review manuscripts that we receive and um 85 percent of the systematic reviews we receive in the journal we were having to send back for at least revision prior to peer review so it'd be like either to redo the risk of our assessment or reanalyze it or do it at all so that was happening with 85 percent excuse me of the submissions that we received so again it shows that you know more than four times out of five uh we've got uh authors who are finding the critical appraisal stuff systematic review by challenging so um one of the big challenges and this is a focus on on the problems because obviously there are other problems with critical appraisal but one of the key things is how to account for all of the risks of bias all of the threats to validity that we need to assess in a systematic review for the particular studies of design that we're interested in and as we are aware I think most of us have been involved in critical appraisal there are many tools available um and there are many tools available especially for observational studies there's long lists that you can find if you just google um critical appraisal or risk of bias assessments but the problem is which ones of these are robust which ones are suitable for for use and therefore which ones should we choose and so what we'd like to do in this workshop is to try and help help you along with that process using the feet principles which we will explain um just another one of these quick questions is which critical appraisal tools have you used so far um and if you've used multiple tools please list whichever ones you've used if you've used a self-made tool because no other appropriate tool was available just say homemade it'll be just good to see which tools people have used um in real life homemade yeah so I call risk of bias tool yeah if you have used a homemade tool um maybe you could just stay quickly in the chat why you used a homemade tool was it because there was no tool for the study design or the review question that you had in mind perhaps that might be interesting to hear yeah so I think we found Jeff in the research that we did that people using homemade tools about 10 12 percent of the time maybe roughly or a lot more in terms of this particular sample it's a small sample size we shouldn't get carried away small size yet yeah n equals three so far um so we definitely have at least one environmental health scientist there as well because we the circle risk of bias tool is for uh preclinical trials using uh animal models okay well don't worry if you're feeling a little bit shy or you can't remember what the tool was or you're just too embarrassed because you um invented one for some embarrassing reason I don't know um let's look at um which tools are available and which have been used so here's just an example of a couple of recent studies so one was published in 2018 one was published um last year um that were looking at the range of um critical appraisals tools that have been used and these are just examples from a couple of areas so this particular study was looking at systematic reviews on medical interventions and this particular study was looking at environmental health systematic reviews um and the key thing that that we'd like to flag up here is that you look there's a wide range of tools available um on the the x-axis we don't need to worry about what what they are it's a little bit small some of this text but the key thing we'd like to flag up here is that there's one particular tool for non-randomised studies that is used more frequently than all the others and that is the Newcastle Ottawa scale um and you can see that it's clearly the most frequently was clearly the most frequently used um critical appraisal tool in in both these studies um and another recent study published in 2019 and and all all of these have got references at the end of the slide set if you want to go into detail and read them up and so the references are all there um this particular study looked at a random sample of non-cockran review protocols in the Prospero database to look at the intended appraisal tools that people were going planning to use um and we have a data set here for several years which is always quite interesting but you can see a get yet again for some some curious reason the Newcastle Ottawa scale is right up there as the the most frequently intended tool to be used um another more recently developed tools have started to get a bit of a problem prominence like the Cochran Robins Eye tool but but NOS is very very popular um so that's that's something we like to flag up and so based on this being a very widely used and popular tool we'd like to um in this workshop we'd like to rate this tool NOS against the feet the feet criteria the feet principles which we will explain so um that's just some background as to why we're going to use the Newcastle Ottawa scale in this in this workshop so what we'd like to do today is obviously we will benefit from this so there's some learning objectives for all of us um the first thing is to just understand what the feet principles are so we'll explain explain those and then how do we appraise critical appraisal tools I mean maybe you'll learn some new things here um hopefully you will um we'll we'll dissect the Newcastle Ottawa scale and see if if there's anything there about the scale that we we need to know and apply to other tools um and as a result of this exercise the our intention is that we should come away with a better understanding of the limitations that there are in critical appraisal tools based on this example of a common tool being the Newcastle Ottawa scale so there's some objectives and we can we can look back on these at the end of the workshop and see whether we've we've actually you know reached these or not and there's a little bonus and that is at the end of the exercise um there'll be an opportunity to to contribute to a manuscript with us if you're interested but I'm not going to say any more about that now because um I want you to stay and watch the rest of the presentation so that's a a carrot for you to keep watching okay so we need to know what the feet principles are um essentially they are overarching topic independent core principles that are the necessary features of any critical appraisal so if any one of these principles is is missing or inadequate then the critical appraisal will not appropriately be able to inform the data synthesis of a systematic review so the important point is that these are very high level overarching principles that are just good practice in evident synthesis they're topic independent um and as such they're quite powerful because you can apply them to almost any situation um and what we've done is we've just highlighted them really so these are existing principles of good practice and we've just decided that it's important that they're highlighted so that we are aware of them that we are aware of this good practice that needs to be you know employed in critical appraisal but the other the aspect of this is we've tried to make sure that the feet principles are very memorable we've made this a short acronym feat so that we remember it so make life easier when we come across critical appraisal tools because i think hopefully when you go through this exercise you'll realize that actually they're keeping things a bit simple is more powerful than having long long checklists we've got so many checklists in evidence synthesis um we could make this into a massive checklist exercise looking at critical appraisal tools and we we'd be confused this hopefully is to simplify it a bit and make it memorable and intuitive but but if it doesn't work we'd be interested to know so um you know it's it'd be interesting to get your feedback as well so the feet principles have been introduced in this paper which we published in environmental evidence last year which we also included in this paper we also suggest an overall framework for how to conduct critical appraisal which is based on these principles so this was a joint exercise with a number of people including Paul and myself are involved but also many other people developed this over over several years it was more than five years it took us to to thrash out ideas and put this together so so that's the the source the key source paper if you want to find out more about the principles okay so that's that's where we are with what the the principles are um are the principles being used by anybody they're quite new we only published the paper last year and the answer is there is there is some optimism that they they're useful they're being taken up by people so so the the feet principles have been included in the latest version of the collaboration for environmental evidence is guidelines and standards for evidence synthesis and environmental management so so um you'll be able to find the the guidelines and standards there sort of incorporate these three principles and these guidelines and standards in turn informally author guidelines for the journal environmental evidence the the feet principles have also been incorporated when Paul was working at environmental into environment international at the journal in in in the triage of systematic review submissions and also in the new evidence based toxicology journal which Paul is the editor in chief of and also in a tool for triaging editorial submissions for peer review relating to evidence synthesis is there anything else you'd like to say on that Paul about those sources I think it's it's a slightly deceptive end right because it's like one of the developers of feet is using it everywhere but I think one of the things I really like about feet is how how much it helps me when I'm editing and I'm you know triaging manuscripts and things uh to to understand how to have a framework that helps me develop appraisal tools of various different types the various different tasks I have to do as an editor the different types of manuscript I receive it's incredibly helpful so it continues to be helpful to the editors of environmental international now after I've left built into a number of critical appraisal tools um and I think that's really what I need to say on that yeah okay thanks so so that's where to look out for the principles um so let's explain what the principles actually are and so there are four principles and they form the acronym feet and the reason they form this acronym feet is to make it memorable so that we can be looking at um a critical appraisal tool we we can you know it's intuitive to know what to look for like we find that really helpful um we find the idea of having a very long checklist with hundreds of questions not easy to work with in this kind of complex situation simplification is the buzzword we like if it achieves a useful output so that's why we've kept this down to four principles that are really important we think so the first principle focused is that the critical appraisal exercise must be focused on a particular construct of interest so what we'll be talking about in this workshop is specifically um internal validity that is a really important construct that we need to include in the systematic review but because the feet principles are very high level overarching principles you could include almost any constructs that you want but in a systematic review we're particularly interested in internal validity so that's the focus principle there has to be a clear focus of the assessment the next principle feet is about the assessment being adequately extensive so having decided that we are uh we have a clear construct of interest we then need to make sure that we assess all the relevant components of that construct it doesn't make sense to assess a construct and then leave half of the components out you know um have gappy data and misinformation so extensive is important and as we'll see in this workshop this is potentially the most challenging um part of critical appraisal and systematic reviews making sure that the the appraisal is adequately extensive when regarding internal validity assessments and then the next one of the feet principles is that the principles need to be applied so in other words you do can critical appraisal and you've got to then use that the critical appraisal in some way otherwise it's a waste of time doing it and the the applied principle states that the appraisal must inform the data synthesis in an appropriate way and it may sound blindingly obvious but when you look at um many systematic or so-called systematic reviews and meta-analyses in the literature they don't they don't even have a critical appraisal step um but from the perspective of critical appraisal tools the output of the tool has to inform the data synthesis in an appropriate way and then the final principle t is all about transparency the appraisal must be transparent I mean it goes without saying almost um this is this is almost a very high level principle being transparent but clearly explained and justified decisions are crucial in critical appraisal as with all steps of a systematic review okay so those are the four principles FEAT feet and I'll just go through them again now quickly um just to reinforce them because these are as I say probably new to you and you won't instantaneously remember them so so here we go focus just a reminder the appraisal must be focused on the construct of interest in a systematic review this is going to be internal validity now you could be interested in multiple constructs you could be interested in whether the whether a study follows ethics guidelines or whether a study follows specific regulatory standards um whether the study is well reported um that's fine you can assess those constructs but you must um in a systematic review also have a sense of the internal validity so you must assess that construct okay so that's an important focus we would say in a systematic review to assess the internal validity of the studies that you include in the review so what kind of issues could potentially come up here is is a question worth asking when we have critical appraisal tools well one obvious one is that you have a critical appraisal tool that just doesn't actually target internal validity so that's a tool that may um may say it's a risk of bias tool but doesn't actually measure risk of bias that's a potential problem um another problem is that a tool may be described as a quality assessment tool which is all very well but what does quality mean um does that mean it's focusing on internal validity or something else um and then another problem that we could come up with with regarding the focus of a critical appraisal tool is that it might just be mixing up multiple constructs in one output so that's not a good idea because we really do need to know about internal validity when we come to our data synthesis and if it's muddled up with reporting standards and other aspects of study conduct that don't have an impact on internal validity in other words systematic error then that gets confusing so so that's focused um and we'll we'll get some experience looking at the focused aspect of the Newcastle Ottawa scale and when we try that out in a moment the second principle is that the appraisal should be extensive um meaning that all relevant components of the construct of interest have to be included so in other words if we're interested in internal validity we need to be sure that all types of bias that could be relevant to our study design of interest are captured in the appraisal now that is actually quite a challenge unless you happen to be an expert on bias and an expert on study designs this can be quite difficult so that's um something to think quite carefully about but nevertheless it has to the tool has to be able to be extensive in order to appropriately inform the data synthesis so the sort of issues that might come up with regarding the extensiveness of a critical appraisal tool could be that missing that certain types of bias are missing entirely so we are interested in internal validity we've got a critical appraisal tool that claims to measure the risk of bias but actually it doesn't contain all of the questions that you need to assess all of the types of bias that you expect to find for that study design and we know for a fact that some environmental systematic reviews just don't include attrition bias at all in their assessments attrition bias is about whether they're missing data this has just been missed out of a number of systematic views so that's an example of you know where the extensive aspect of the critical appraisal is compromised as I said it's actually quite challenging to know what types of bias you would expect to find in a study and so this is hopefully going to be something that we get a bit of a grip on and this afternoon the third principle applied obviously as I said the appraisal must inform the data synthesis in an appropriate way so it's got it's got to provide an output that is useful to inform the data synthesis so that's using some kind of appropriate output scale in a way that then can be operationalized in the data synthesis such as inner sensitivity analysis or subgroup analysis or some kind of bias correction there are various different ways that you can actually incorporate internal validity assessments in the data synthesis provided that the critical appraisal tool you use gives you the right output so that's what the application is about and some of the issues that could come up with the application is that different types of bias could be mixed together in one scale and within a particular type of bias the rating scale that you use to judge the the risk of that bias so for example a low risk or moderate risk or a high risk that scale may be misinterpreted so it's it's an ordinal scale but it might be interpreted as continuous so then you end up with people calculating numerical operations and statistics from it which is not appropriate another issue with the output of critical appraisal tools is that sometimes they recommend the use of summary scores which on the face of it sounds like a good idea because bringing things down into simple convenient numbers is quite appealing but the key problem is we get that we get with summary scores is that they can seal useful information they you lose information on where the problems are in a study so if a study is subject to multiple risks of bias and you end up with one summary score that doesn't separate them then you lose useful information about how to interpret that study and how to you know progress science if you don't if you don't know what the limitations of science are so those are the first three principles and the final one is that the critical appraisal tool needs to be transparent or the whole critical appraisal process needs to be transparent but by definition the tool itself also needs to be transparent and the issues with transparency that can come up with are that you have a critical appraisal tool that doesn't require you to record and justify your decisions this is important because there's a lot of subjectivity in critical appraisal and so transparency about how you reach your decision and how you decided that there was a risk of bias in this study is absolutely crucial and sometimes the output of critical appraisal tools isn't particularly intuitive the link between the questions in the tool and whether there is a risk of bias or not it's not very clear so you may have a yes no output to questions in critical appraisal tools what does that actually mean in terms of the risk of bias and we'll have a look at this when we when we come to the Newcastle scale in a moment so that is a quick summary of what these four key principles are about making sure that we have a clear focus so in our case we're interested in systematic abuse so that's internal validity we're making sure that we are adequately extensive in capturing all components of internal validity so we need to make sure we have all the sources of bias accounted for that are relevant to our study design of interest and then we must make sure that the output of our critical appraisal can inform the data synthesis appropriately and then that the whole process is adequately transparent okay so we're going to now have a look at the Newcastle lot of a scale against these three principles to see how well the nos does and just to remind you the reason we've chosen the nos newcastle to a scale is because it's you know it is the most popular critical appraisal tool for observational studies so the first thing to say I just mentioned that we've we've assessed the selected the newcastle to a scale there because it's the most widely used tool the newcastle to a scale has actually got two versions one is for appraising cohort studies and the other is for case control studies now just to keep things manageable this afternoon we're going to focus on using the cohort studies version of the newcastle to a scale as an example now the newcastle to a scale tools are available from a website which is here and at that website you will find a slide presentation which briefly goes through the tool and how to score the tool and an example of the output which we'll show you in a moment there's a word and pdf version of the manual and there's a word and pdf version of the checklist or the newcastle autobus scale checklist so this is all available from a website and and that may maybe is why it's widely used it's readily available readily accessible so the scale itself is a long checklist oh i say it's a long it's kind of slightly annoyingly long to not quite fit on a slot powerpoint slide but it actually only has eight questions so it's a checklist with eight questions and it's divided into three sections so there's a section about selection comparability and the outcome in a cohort study and in a moment we'll just have a quick recap of what a cohort study is for in case we're not clear about that there are two or three statements per question relating to study design and you essentially need to choose the statement that best matches the study design that you are interested in and then the the answer to these questions comes in the form of a star and questions can be awarded a maximum of one or two stars depending on the question so selection questions have a maximum of one star the question on comparability here has a maximum of two stars it can be awarded and the outcome questions have a maximum of one star that they can be awarded and the overall output of this this sort of scoring exercise with stars if you like is to give you an overall assessment of the study quality okay so that's how this Newcastle Ottawa scale checklist for cohort studies works so let's just go back and remind ourselves about the feed criteria and have a look at whether or not the Newcastle Ottawa scale for cohort studies does align with these core principles for critical appraisal so this slide looks a bit cluttered but it's essentially what we've got here we've got the eight questions here divided into the three sections which I mentioned so selection comparability and outcome now there isn't time really for us to sort of go through every question one by one and start discussing all the possible sources of bias or threats to internal validity that could occur but if you were to look at this scale this checklist in detail you would see that all of the questions here relate to certain aspects of internal validity so they are valid questions they are legitimate questions to ask about different types of bias that could occur in a cohort study now you might not want to take our word for that you're very welcome to go away and look at this in more detail but we're illustrating how you would go about assessing a critical appraisal tool against the feed principles so that's what we're trying to illustrate here there is a slight deviation here in the first question in the Newcastle Ottawa scale for cohort studies asks us about the representativeness of the exposed cohort which is actually really a question about external validity so it's whether the cohort in your cohort study matches the wider population of interest or relevance to the review question so strictly speaking this is a question about external validity not internal validity now you may decide that that's not a problem because external validity relates to a systematic error and needs to be assessed and as does internal validity but that's just an example of where the Newcastle Ottawa scale doesn't quite totally cover internal validity so in all the other questions we believe that internal validity has been assessed but this question says this is external validity so it does pretty well on the focus part of feet so we get a fairly big tick for the F I think there's certainly nothing really alarming going on here so that's how you rate a critical appraisal tool against focus is it focused on internal validity yes largely yes so what about the extensive component now that we know that the Newcastle Ottawa scale does address internal validity does it actually capture internal validity adequately now in order to answer that question we need to take a step back and just remind ourselves what a cohort study is because what we need to do in order to answer the extensive question is to decide whether or not all of the possible sources of bias that could come up in a cohort study are captured in the tool and this is where things start to get a bit complicated and fiddly and you know there's no escaping this it's it's just life that this is how critical appraisal works we do need to have some knowledge of study designs and risks of bias so excuse me while I go one step and remind us what the what a cohort study is so a cohort study is a study where we have essentially one population that we are interested in following to understand whether outcomes of interests occur in individuals in that population who are either exposed or not exposed to a particular intervention or exposure situation so you're following basically a population through time so that's essentially what a cohort study is it's following a population through time there will be exposures in that time period and then the time period ends with the population having or not having the outcome of interest that's the simplest description of a cohort study now just for for a reference it's useful to have a kind of question in my read systematic review question in mind when thinking about study designs so the Newcastle Ottawa scale website does actually pose a question a systematic review question so we've gone with that just in the illustration slide here so the question is asking whether or not cardiovascular events differ between post menopausal women who have received hormone replacement therapy or not so that's actually a pico type question and it's very amenable to a cohort study design so so this slide is really just to remind us what a cohort study is so that's the first thing we need to be clear about before looking at critical appraisal tools what is the study design that we we are including in our systematic review and you know is the critical appraisal tool appropriate for that just to mention that cohort studies can be prospective or retrospective in a prospective cohort study you recruit a population and follow that through time in a retrospective study the population has already been recruited and followed through time in history and the data are available in some place like a database or a patient registry and you're going back to those data um i'm just mentioning that because there are different risks of bias in prospective and retrospective studies um so that that's something to think about um so anyway we've got the Newcastle Ottawa scale which claims to um be useful for cohort studies we already know that it's uh it does target internal validity which is good and so what we need to know now is does it cover all the kinds of bias that you might expect to find in this kind of cohort study um so this is where things get you know get get interesting and this is where you know you might might want to go and meet with colleagues and have a coffee and say well what are the kinds of bias that you can find in a cohort study so a cohort study could suffer from different types of selection bias so for example if the cohort is not representative of the target population actually this is something that we call external um validity in the um example just now i think this jeff would be relating to sampling more than uh so whether or not the sample for your target population is yeah representative but but the key the key point here on this slide is just to to point out that there are actually multiple different sources of bias that can occur in fact quite a few when you look at it in in a cohort study so um the the cap the population characteristics or the baseline characteristics of the two cohorts the um the exposed and unexposed cohorts of people within the population of interest um they they need to be similar otherwise a confounding variable might explain the outcome so you know obviously if one the exposed cohort has a different age to the unexposed cohort then that's a confounding factor that might explain the outcome rather than HRT which is what we're interested in so that's an example there could be missing data between these cohorts that's not balanced so that's a risk of attrition bias there's a possibility in cohort studies um that the exposure condition is misclassified it's not correctly defined um and then you've got risk of biases in detection of the outcome so how the outcome is measured um and these will differ according to how what kind of outcome so with in this example with coronary heart disease um you could be interested in measuring mortality due to coronary heart disease you could be interested in whether people experience angina you could be interested in whether you've got radiological evidence of narrowing of the coronary artery there's different kinds of outcome that would all need to be checked separately for their risk of bias and then outcome reporting bias is also a potential problem so having got an outcome has it been properly reported so you know if it's a continuous outcome it's been reported as a continuous outcome not sort of spliced sliced up into dichotomous data or ordinal data or something like that so there's different ways that outcomes can be reported that can introduce bias that might apply and then the partiality and the overall analysis does do the people doing the overall analysis like the statisticians do they they are they aware of the allocation groups of participants um or or essentially who was exposed and who wasn't exposed so did the statistical analysis differ in any way between exposed and non-exposed groups so these are just examples of the the things you need to think about and some of these are very familiar types of bias selection bias we come across a lot and some of them may be a bit more specific to so these types of cohort study for example misclassification of the exposure so these are things to keep in mind when we look at the critical appraisal tools that we are interested in potentially using are all of these types of bias going to be covered so just have a very quick question in the chat that seems quite relevant actually so we don't we don't need digress too hard but Stevie has asked uh you know he says that he doesn't have the expertise to define all possible sources of bias that's fine none of us do right um he doubts that many researchers have come up with all relevant possible sources of bias so what sources or tips do we have to be comprehensive in doing the bias kind of defining the the domains or types of bias that we should be concerned about yeah um there's a really good question um and if we had a perfect answer to that question we could all go home and enjoy ourselves um one of the reasons we got this workshop is because we don't have a very good answer so it's a little bit like a chicken and egg situation isn't it we are here we're talking about selecting a critical appraisal tools and if you had a really good fit for purpose um critical appraisal tool that was based on causality theory and empirically supported in that way you might trust that tool to guide you on all of the types of bias that you need to know in which case you wouldn't really need to worry too much when you join a systematic review team you'd be given this guidance from the tools that might be the case for certain tools like for example those developed by Cochran are very strongly based in empirical theory of causality so you know these tools do give us some confidence um once you look at randomized control trials we do have strong confidence that we know most of the types of bias that they're likely to be susceptible to once we go down the list of you know the hierarchy of evidence in some of the observational studies area and especially if you get hybrid study designs where you get mixed um types of observational or even quasi experimental studies um it gets much more murky what we would recommend I think and this is this is where it might be worth you having a look at the paper um in environmental evidence where we introduce the feed principles we recommend a kind of systematic way of working through this when developing a systematic review protocol so one of the things you can do is you can um you can have have a stakeholder engagement exercise with topic experts and possibly experts in causality theory such as statisticians or and statisticians if or mathematicians um so stakeholders engage with stakeholders to work out whether anybody has spotted a source of bias and what confounding that hasn't been found it can be very helpful to develop conceptual models so you can sketch out what the what is the dependent variable what is the independent variable what is the exposure pathway um sketch that out um and discuss that send that to your advisory group your stakeholder group use that as a focus of discussion even put it on you can eat you know if we had the time you could even put it out for peer review um that could be that's one of the functions of developing a protocol and having it peer reviewed in a systematic view the other thing that you can do is to an extension of conceptual models is to look at directed acyclic graphs which are a way of formally investigating um the relationships between dependent and independent variables and where confoundings and clearly specifying where confounding could occur okay so there's a sort of structured process of trying to make sure you covered your basis if you like um and if you go through that process of consulting with experts making sure you're clear what your conceptual model is what your logic model is for your systematic view question um and making sure that you you haven't you don't think you've missed any types of confounding then your systematic review report is likely to be quite strongly defensible um but clearly even having gone through that process there might be sources of error that nobody's thought of that would be missed um but that would definitely be an improvement in the sort of level of rigor that that um people currently conduct I think from any systematic views I don't know if that's answered your question enough I mean I wish I could give you a perfect answer is anything to add Paul yeah I think so I mean there's all of that um you know we're as a naturally incredibly lazy person uh there are also some tools that have barely robust development processes behind them so you couldn't be blamed for example for taking robin's eye or robin's e if you're um looking at um non-random my studies of interventions or exposures because that's been well worked out over like 15 to 20 years of experience by the Bristol set right in heaven space medicine there's not going to be a huge amount that's missing from that and in using tools like robin's like per protocol um they would say you'd have to sit down and carefully work through the potential confounders and things with um like a an expert group of epidemiologists working in that topic space and then you're pretty good I mean that this stuff is really tough because in some domains these issues are quite well understood and there are tools that are quite well established and that work quite well um that the challenge that I think one of the challenges that Jeff and I are trying to address in developing the feet framework is the um the well developed tools tend to be rarely used and there's a lot of use of tools that are less well developed so what we're trying to do I think if it's what if you apply the feet framework and you kind of you know you you can quickly see there are issues in Newcastle also that with issue the Newcastle Ottawa scale and you apply the feet framework to something like robin's you'll see that the the issues with Newcastle Ottawa scale just aren't there with robin's and there's no such thing as a perfect assessment or a perfect tool uh it's about understanding what the tool does and its limitations within the context of the systematic review we're trying to do you have a good handle on that you've done a good job basically great good answer okay thanks for the question that was great um it's quite challenging to cover critical appraisal in a in a one and a half hours workshop you know we it's amazing we can actually address in a pool given how complex it is actually on the subject of that Jeff we've got 30 minutes that's fine I think I think it's fine um so just to recap because we did a slight diversion there um to answer the question um yeah so we basically we have got to the point where in order to understand whether a critical appraisal tool is adequately extensive to fully capture internal validity we need to know what the internal validity issues are in that in the study design that we're looking at and so we just as we just discussed there's there's a lot of them um so does the Newcastle Ottawa scale address these main types of bias that we would you know not be surprised to find in a cohort study um and uh so the answer in in very quick summary form is it doesn't so these highlighted um types of bias are not either not captured at all in the Newcastle Ottawa scale for cohort studies or they're not fully captured so um didn't you know we could take we could take this scale and discuss it for hours and hours it's um it's you know it's quite a complex area critical appraisal but but basically um we've kind of we've looked at each of these questions um Paul and I have looked at them and decided between ourselves that we don't think that these um are um or we do or we do we do what we don't think these are actually um relating to um bias i internal validity um and the output of our assessment is that yes um selection bias is well covered um unbalanced missing data that's reasonably well covered misclassification of the exposure or outcome does appear to be covered and detection biases appear to be covered um some types of confounding are covered um in in the questions um but there aren't any specific questions that would uh address bias in the way that outcomes are reported and there aren't any questions that would check whether there's any um partiality in the analysts um of the study so in this sense um in in our assessment the Newcastle Ottawa scale does actually miss some key some key types of bias that we would hope to have um assessed a study for a cohort study okay so um you know already even though this might not be an exhaustive list of biases that we started with we can already see that someone some types of bias that we we would expect to be to be assessed are not assessed by this scale okay so that's that's how Newcastle Ottawa rates against the extensive principle so let's look at how the Newcastle Ottawa scale for cohort studies rates against the applied principle so um this is taken from the Newcastle Ottawa scale's website um um the slideshow presentation at the website and uh and it's an example of we we think how the Newcastle Ottawa scale developers um anticipate that the tool output would be used so um the output um is a summary from the star answers to the questions for each of the categories of selection comparability and outcome sections of the checklist um and you can see that uh in this particular review that this is based on um there were 14 studies 14 cohort studies and so this is a this gives us a summary of how each of these studies compared um regarding these three sort of sections three domains of the Newcastle Ottawa scale um but there are some key things to note here um obviously when you remember back to how we answered the questions in the Newcastle Ottawa scale some of the questions were allowed to have one star maximum answer um and the comparability question um was the only one that was permitted the maximum of two stars but there was only one comparability question and there were four selection questions so you can immediately see that um these are not directly comparable against each other in terms of the kind of quantity of stars they within each category certainly there's a relative comparison you can make on how the studies perform but you can't compare across categories um and so uh that's something to think about in the in the output so how can we apply this in a data synthesis does this can this appropriately inform data synthesis and this is another example from the Newcastle Ottawa scale website itself and they've suggested that you can literally tabulate these star outcomes against um your data synthesis output which in this case is a standard forest plot and this looks actually like you know it's quite informative it does tell us that um there are differences in aspects of rigor between these studies and as we already know that the Newcastle Ottawa scale does target internal validity we might actually go so far as to say well there might be some differences in internal validity between these studies um shown by this output the problem is that as I said these are not comparable scales because they don't they're not they're not um you know numeric scales in the sense you can just add up the number of stars um so you can't just for example say that the Laritzon study has seven stars and therefore it's better than the Palettini study which has six stars unfortunately this is exactly how people use the output of the Newcastle Ottawa scale in most cases as far as we're aware so um the scale could potentially have a maximum number of nine points if you regard a star as a point um and people have used this to come up with a scale of one to nine um to then try and interpret that in terms of some kind of um you know risk of bias interpretation which doesn't make much sense what what is um you know is this actually useful for the data synthesis it's partly useful in that we we can see some patterns but there's a real danger that you're going to end up summing these up um and giving an output that has nothing to actually do with risk of bias so that's um not a fully appropriate output to to see for a critical appraisal okay um so question we really want to know if we're going to conduct data synthesis is how susceptible to bias is each study so just for example we've got a Palettini study here um um and the Lafferty study um we can see from the way these stars are organized on this against this forest plot that the Palettini study has some issues with selection and outcome related biases whereas the Lafferty study doesn't have those issues but it does have some other issues with comparability of the cohort um and as I mentioned you know the score the adding up of scores is the real problem with this type of output from a critical appraisal tool so you might think that Palettini scores six out of nine compared to Lafferty eight out of nine um that's just doesn't mean anything in terms of risk of bias does it so we do need a different way of rating the output from this particular tool now it's not to say that the Newcastle Auto Scale is a disaster I mean you know it does assess some key parts of internal validity but it maybe it needs to be modified so the feet criteria could actually be quite useful when looking at critical appraisal tools to identify where the tools might be deficient and might need to be modified um as long as you have the permission to modify tools so certain tools might have copyright restrictions we have to be careful of but but um you know you could potentially form a new tool from the new car from the good parts of the Newcastle Auto Scale so anyway finally um is the Newcastle Auto Scale transparent does it meet the basic principle of transparency uh no unfortunately because the tool doesn't ask us to um provide any rationale for our judgments that we make so um it might ask us whether the cohorts are comparable on their baseline characteristics but if we just say yes or no or select a star that doesn't tell us why we made that decision is it because they differ in age is it because some of them um uh have a different um disease history you know that information could be crucial for understanding how to interpret the data and it's just missing so that's why recording judgments about risk of bias decisions is so important and this is something that the Cochran tools do require people to do so just to um before we have a sort of we finish off and and open up for a discussion um it's worth saying obviously the feet principles can be applied to any critical appraisal critical appraisal tools we've just used the Newcastle Auto Scale as an example because it is so widely used um but uh there is an example in the environmental evidence paper um where we've applied the tool to the JBI critical appraisal tool for cohort studies and um another tool called the OHAT risk of bias tool for cohort studies these are just examples um and you can see that an overall summary um in this table shows that these tools generally do target internal validity quite well so that's not really an issue but they start to fall down on whether they adequately capture internal validity so you know we're really interested in risk of bias but actually these tools aren't capturing all of the risks of bias the with the exception perhaps of the OHAT tool which does seem to cover most of the the types of bias that we would expect um but you could go away and look at OHAT you might disagree with us given that we've had this discussion about how can we ever be sure about this so that's you know that's open to some interpretation um the application of the tools varies so the final rating we've just discussed with newcast a lot of it is a total score which can conceal important differences between the studies um in the JBI tool for cohort studies um the output is a series of yes no or unclear answers but these don't directly translate into internal validity they don't directly translate into yes there's a you know there's a high or a low risk of bias that you can immediately operationalize in the data synthesis um the OHAT tool is better um it's got some stronger signaling questions that seem to link through to a logical risk of bias judgments um and as for transparency again the similar problems that we've talked about um Newcastle Ottawa and the JBI tools don't don't expect the review team to record any judgments about why they decided a study was at high or low risk of bias or not um whereas the OHAT team tool and as I say the Cochrane tools as well do ask for this information so that's just a summary table just to show you how you can kind of you know rate the the feet principles um against these tools or rate these tools against the feet principles um and hopefully this table gives you you know a feel for why having these four principles just saying yeah focused extensive applied transparent these are four key things to look for um it's quite powerful rather than having a like 90 question checklist that goes into all the the bowels of critical appraisal which you could also apply um to assess critical appraisal tools um so to recap um I hope that this has given you an understanding of the feet principles why we've come up with them why we what why I mean the principles already exist we haven't come up with the principles but what we've done is we've raised them as a short memorable acronym so that it makes life easier that's basically what we've done um so do you understand the feet principles if you don't understand please pop something in the in the chat thread and we can pick that up and hopefully this little exercise just focusing on the new castle Ottawa scale for cohort studies is a good illustration how we have to be careful when we come across critical appraisal tools which we shouldn't just take them up face value we need to critic you know do a critique of the critical appraisal tool really um that's something that you would want to do in the protocol development stage of a systematic review um be sure that you have selected appropriate tools so hopefully today that's helped you to be a bit more wary perhaps of critical appraisal tools um and yeah and also to understand this you know where these limitations are in relation to um the the types of internal validity threat that we would would be interested in looking at so hopefully those learning objectives we can tick a put a tick by but if you feel that you haven't you know understood what we're talking about or there's something deficient that you'd like some clarification or do put something in the chat so we can pick that up um so now there is an opportunity actually I don't know how much time we've got left Paul are we okay got 20 minutes and we have a few questions that yeah so questions we can dig into so brilliant brilliant so we did I did kind of rush through that reasonably quickly just to make sure we didn't have no time left at the end um shall we mention before we take questions shall we mention about the possible involvement in future activities what should we do that after yeah no let's do that now because like we've got this if you remember this is a long time ago now but if you remember at the beginning we said there could be an opportunity for you to contribute with us to a manuscript if you're interested um so we've basically we've explained what the feat principles are um so we've tried we've tried to explain them to you and I don't know how well we did maybe we didn't do a particularly good job or maybe it was okay but but we think that probably it could be beneficial to produce a guidance document or a sort of user guide or or manual to to to publish to explain what the feat principles are to people and so obviously get some uptake so I mean as Paul said it's a little bit biased um the way these principles are being used at the moment because they've been adopted by the collaboration for environmental evidence which I'm involved in and the journals that Paul's been involved in so it's we've got a bit of a vested interest in those examples but you know if we if we feel collectively as a workshop that these are useful principles that are worth disseminating then probably we need to disseminate them further than we are here um with something a bit more um explanatory than we put in the environmental evidence paper so the environmental evidence paper that I explained about is covers the whole of critical appraisal how to plan and conduct critical appraisal it's it's got a fairly small section explaining feat principles although they are important so there could be a a role for a say user guide or or a guidance document or something and this is where we might want to consider um you know sub questions within feat whether we develop a bigger checklist to to more appropriately make sure that people don't miss things that are relevant to feat the danger of course is that we end up with something like a consort list or a prisma list which goes on forever and people you know just don't want to fill in or don't want to use because you know I get allergic to checklists and that's why I wanted feat to be so intuitive and simple so you keep it in your head almost but is it useful to have a checklist to support it that you know maybe optional or to make sure that um the items are all applied appropriately I don't know this is an open question this is where we really like your feedback um and if you're interested and if you'd like to join in um putting something together having a discussion around um where to go next so we would organize some some meetings teams or zoom or google meet or something wouldn't we pull I think to pick this up yeah do you want to say anything more about this I just think that um this is an ideal opportunity so people who are new to this uh we're still learning to talk about it and how to write about it and how to teach about it so having people who've been on the receiving end of is this our first proper feat presentation that we've done yeah yeah pretty much so I think everyone who's in this call is ideal uh potential co-author just to help shape that kind of guidance manuscript and your contributions be really welcome so just drop Jeff or I an email afterwards and uh we'll uh get back to you about when we're going to have our first call about it and start poking something up yeah so there are our um details and just before we before we talk about um you know pick up the chat just to let you know that at the end of this presentation are the references so if you're watching this retrospectively on YouTube um you can scroll down a bit further and you'll see all the references for everything we've said above so Neil um sorry Neil not Neil Paul um or Neil in fact whoever's monitoring the chat anybody um what are the questions so I've got the chat in front of me we've got um an interesting question about rapid reviews but I think we'll just loop back around to that because it opens up a broader issue I think than what feat is necessarily designed to address someone asked if we've already identified tools that tick most are all of the boxes of feat there are quite a lot of quite good tools out there so I dropped in the chat tools like robin's i robin's e rob2 us ntpo hat that was in one of our examples the navigation guide in the environmental space have perfectly serviceable risk of bias assessment tool the original higgins 2009 rob tool is you know I think the reasoning that went into rob2 is is valid and sound but it doesn't invalidate the previous generation of the cocker and risk of bias tools uh and now the actually reminded me about it so the gianna briggs institute tool that you had in your slide they updated it since we wrote the um christian appraisal papers together and the new version of the gianna briggs institute which I think came out maybe like a maybe it was recently six weeks ago eight weeks ago or something uh it actually addresses a lot of the issues that we would identify through feat in that original jbi tool and the new tool actually resolves a lot of those issues does it that's interesting because I I very briefly tried to look at the new tool because I was aware it had been updated and I the version I looked at which I thought was the updated version um still didn't address some of those issues so I think I must have accidentally looked at the old version think they have a lot of tools so it might be that they've done a better job in addressing the issues in the rcts tool than they did in the cohort tool that I've not looked at them in detail I just had a quick browse through and I was like you know this is looking much stronger than they used to anyway right so when I looked at what I thought I'd done um was look at the latest version of the cohort tool and I thought if I'd done that correctly um that it hadn't really addressed the issues but this is something that we can all go away and look at so this is a nice little exercise to go and see whether the latest jbi tool actually does address the feat principles so don't take our word for it um have a look maybe good good little exercise so then Matthew pulled a good question from slack which I didn't see uh so someone says again this is more of an issue relating to publishing practices as much as anything else so they said that they think one of the main limitations in the feat framework uh is incorporating word limits in journals and they say that in their experience of systematic review publishing they struggle with word limits so when they want to fully discuss problematic risk of bias domains in their included studies they're basically not allowed to buy the journal and speaking as an editor I think this is just one example of how badly wrong we're getting publishing with word limits right it's the internet we don't publish on paper anymore there is no reason to have word limits of journals every journal I've worked at has abandoned word limits for systematic reviews the moment I walk through the door my new journal has no word limits on everything anything at all there are no figure limits it's completely and utterly absurd so if you've got journals requiring stupid stuff my advice is publish somewhere else that's how you solve that problem yeah yeah yeah the doubt the downside of having no word limits of course is if you're a peer reviewer you can be sent a systematic review that's 85 pages long with 12 something that's also so just to give the other side of stupid right there are some journals that are totally stupid and don't require authors to be concise which is also a problem so the guidance for my journal and sensible journals is uh users with many words you need but be as concise as possible right so you evaluate it from both sides yeah so if you get an 85 page systematic review obviously a different mistake has been made but um yeah hard word limits are ridiculous yeah so it's the other way around isn't it it's not that the journal the problem with the feet principles has been identified it's a problem with the journal that's been identified positive principles so then we've got so Heather Schafer uh says that uh they find that environmental health research has considerable pool of cross-sectional studies do you happen to know if Newcastle Autoscale or others have strong tools specific to address the biases of the use of cross-sectional studies and how to use cross-sectional studies in an applied context considering these biases uh Jeff you can have that one well the simple answer is um if i i can only answer this in a way that how i would approach it because i i don't personally know of a tool that i would say is very strong in this area um now that doesn't mean there isn't a tool so please don't take my word for the there might be a tool out there um i would systematically consider um based on my own knowledge of study designs and sources of systematic error i would work through what um type of cross-sectional study i am conducting um and i you know try and identify what types of of bias i would expect to need to consider and then i would look for some you know i would do an up-to-date search on the latest cross-sectional study tools of which there are actually quite a few that claim to cover cross-sectional studies Newcastle Autoscale um i don't think it strictly covers cross-sectional studies although there is some wording in there that implies that you can use the case control tool for cross-sectional studies i'm not quite sure um but jbi jbi tool the jbi has one for cross-sectional studies and so on what i would do is i would try and get get some idea of what i'm actually looking for in my cross-sectional study so um once i'm clear about that i will then look for tools that match um based on that sort of stepwise procedure that i talked about so that's make sure that i personally am confident that the types of bias that i'm expecting that could come up um i can assess i would hopefully if i have time ask other people other experts um send my ideas around or ask the rest of the team if it's a project with us an advisory board ask them um and then if necessary you know this is the point at which you might potentially identify the need to make a new tool or a bespoke tool or adapt an existing tool so for example if the jbi tool for cross-sectional studies doesn't cover some aspects or bias that could arise from your specific setting for that cross-sectional study then you might make a case for you know using that tool with some adaptation add some extra questions in something like that um we cover this systematic so structured way of dealing with this problem if you don't know if the tools are adequate or if they exist to properly address your question we cover this systematic sort of process which i mentioned you know stakeholder engagement conceptual model directed to sake cyclic graphs looking around at the different tools that's in the paper and environmental evidence we just we just list those steps there um so now you caught you caught me out because i i don't um i don't know of a specific tool that i would say off the top of my head is yeah this is the go to not like we the ones that paul has mentioned for for other study designs so paul you mentioned you'd be quite confident using um robin's i robin's e of rob rob two oh hat things like that i can't say the same for cross-sectional studies yes the reason i gave you that question is i didn't have a good answer and i'm just kind of bummed preserve my reputation by saying nothing at all no but this is this is a really good example this is a really good example of where we can go away um no i i probably need to do this now because you asked me the question i probably need to go away with the feet principles and actually look at a few um cross-sectional study tools and see which ones are good because this is a really good question i i didn't really know what the answer is to that without cross-sectional studies are tricky to deal with anyway because you're measuring exposure at the same time as outcome typically so um there are ways of designing them and certain types of exposure and outcome that they're better suited to understanding and then types of outcome which might have exposure which might be very short-lived where the present level of exposure isn't indicative of historical levels of exposure would be more challenging right but um it's like any so you have to take it in context absolutely that that is a great example of the kind of you know um discussion that you would have with expert stakeholders um you know so for example if i want to conduct a cross-sectional study i think i'll i'll have a team's call with you and then you can tell me all about those issues you're just talking about you know that's that's a good part of the process okay so we had the question about rapid reviews because again i think this segues nicely because we're talking about context suddenly and when people say is it justifiable to do such and such then it largely depends on context right do we got the good reasons for doing so so um i'm just trying to find the question now so uh joseph language asks if one is undertaking a rapid review being extensive may be difficult is it justifiable to accelerate the critical appraisal process by leaving out certain types of bias and therefore compromising extensiveness that is a really good question um and it's it's it's a it's a very general question about where do we need to do critical appraisal in the spectrum of reviews so you go all the way from the fully fledged systematic review which is um very intensive um but designed to give you a very reliable output down to the other end of the scale where you need to do a rapid synthesis and the answer to that question depends on why you need to do a rapid synthesis so you know the very far end of that rapid scale is like you need to do and you need an emergency answer to a question in two hours if you need to do a review that's that rapid then clearly you can't do a full critical appraisal you need to just gather evidence together um okay if you are conducting a rapid review that's not quite that far along the scale and you have some justification for doing the review rapidly rather than more systematically then i think ideally you do need to at least consider um whether or not critical appraisal whether it's acceptable to exclude critical appraisal because at the moment most rapid reviews don't even consider critical appraisal so there've been various surveys done um recently which show that critical appraisal is always just essentially ignored in rapid reviews which is a problem if you are doing a rapid review of a fuzzy question that's not going to give you a hard quantitative data point that's important then maybe your risk of bias is not such a strong consideration it depends on what you know how important is the quantitative output of your evidence review is it going to inform a model parameter is it going to go directly into a guideline is it going to inform a policy committee discussion what is it going to do and is it you know in medicine maybe it's really crucial that your output you know how effective is a cancer drug is you really got to be certain about that but it depends if you've got more of a fuzzy policy question um you might get away with not being quite so concerned about some aspects of the critical appraisal so this is intrinsically linked with why you're doing the why you're doing the review rapidly in the first place you've got to be able to justify that and why and what your question is how sensitive is your question going to be to systematic error okay and does it matter if it's if it's if it's not a critical question it might not matter so if that's the case then you might be able to make a case for not doing critical appraisal at all but what I would say is don't ignore it just put a sentence in your protocol explaining why you aren't doing critical appraisal you know it's important to justify your position but the the other thing is that it this is something that we're quite interested in potentially developing further if anyone's interested um is whether something like feet could be a useful framework or checklist for rapid reviews because when you are doing a rapid review as the question correctly you know nail puts the nail on the head hits the nail on the head here um which if you're going to do critical appraisal which bit do you miss out if you've got to do a rapid synthesis and so having a framework where you say well okay we you know critical appraisal is structured um it's got to be definitely we definitely know that we've got to assess the internal validity for this particular review question but can we miss out some types of bias so that's where you're prompted by the extensive question and then the question was about which ones can we miss out well we don't have an answer to that specific question but we we know that we've got to ask that question and we know that in doing you know developing the protocol for that rapid review you will have had to have gone through this talk process and justified that you're not going to assess you know maybe attrition bias and you can put a justification for that because maybe the kinds of studies that you're including in that rapid review don't suffer from attrition bias because of inherent features of the design or something so yeah i think a structure could be really helpful um but it's a really good question does does that answer it in any way anything to add paul um i think that's very well we've got two more questions from youtube that we should try and just just wedge in in the last two minutes and then i just want to also mention something that a couple of people from the questions might be interested in because it relates to how do you catalog bias so first question from youtube is do you ever make inferences on the degree of impact a bias has on the point estimates and the answer that question is it depends because it's very difficult to generate empirical evidence of the magnitude of bias that is introduced by some shortcoming or limitation of study design so it's like in one experimental context um blinding or masking might not uh might have a trivial effect often if the outcome is very objective like mortality then you're not going to worry too much about masking but in other contexts you've got a subjective outcome that's being evaluated then not masking investigators to exposure status could be very very important and then developing evidence for that particular study design how important masking is might be very difficult to do if not impossible so what we generally do is you have some kind of heuristics rules of thumb that we follow where we where we try to understand how worried we are based on the information we have about the subjectivity of the measurement or you know it's the potential importance of something but we wouldn't usually try to adjust a point estimate around potential magnitude of bias because it's so hard to establish whether or not that adjustment would be correct as we get bigger data sets and things it becomes more possible to do some of this stuff but we approach that very very cautiously yeah it's a difficult area um I went to an EFSA colloquium a few years ago and there was a session about bias adjustment methods and I left that session thinking this is nothing to go on at the moment it's it's a very niche area where there are statistical methods available that you can apply but um they're very much applicable to very specific situations um there's no general approach that we can could suggest I don't think at the moment okay so I have 60 seconds left so I just want to quickly answer so Huzaifa Imam an apologies for my pronunciation asks if the limit of NOS is that you can sum up the individual ratings and compare the sums between studies and hence decide which study has a high risk of bias that is one of several problems uh so yeah so generally adding up across bias domains is a bad idea that we have run out of time to explain in detail um but there's some good guidance in the Cochran handbook on why you shouldn't do that um good theoretical reason for not doing and then the final thing I wanted to mention which I think particularly relates to Stevie's question is that there's a project called the scientific evidence code system that I'm involved in where we are running through trying to develop a controlled vocabulary for health research and that includes 140 different risk of bias terms and types we meet once a week on Fridays to do the consensus process for defining these terms so if anyone wanted to be involved in that it is an open prodigal project people volunteer to participate prompt me an email and I will tell you more about it thank you so much for that brilliant workshop I really enjoyed that I hope everyone um also enjoyed it as well I've learned so much um I'm going to definitely contact you guys and and hopefully try and contribute to this project it was very interesting um yeah I hope everyone's enduring the whole process and we'll see you again soon thanks so much bye thank you