 My name is Monica Hildkranz, I work at the Swedish Council for Health Technology Assessment and Assessments of Social Services, and I've been asked today to talk about our experiences of using CIRQL to address our confidence in findings from qualitative evidence synthesis. So I will give you a short overview of what CIRQL is, what assessments are made, and then go into a recent report from SBU where I can exemplify these concepts. Then I would just like to also acknowledge the great CIRQL coordinating team for giving me some introduction here. So CIRQL is a systematic and transparent way of assessing how much confidence to place in findings from qualitative evidence synthesis. It was developed by researchers with a background in qualitative research and systematic reviews, and the idea was to make findings from qualitative evidence synthesis more useful in decision making. And they started developing this methodology in 2010 to support the use of qualitative evidence synthesis in a WHO guideline. And the basic idea is that you address your confidence in each individual review finding and you will for each finding end up on one of these four levels. So you might have high confidence representing that it's highly likely that the review finding is a reasonable representation of the phenomenon of interest. And then you go down moderate confidence, low confidence, and finally very low confidence. It's not clear whether the review finding is a reasonable representation of the phenomenon of interest. So how do these assessments? You're considering four different components. You have methodological limitations, relevance, coherence, and adequacy of data. And I will go through all of these separately. I just wanted to stress first of all that. So here you're looking at each individual review finding. You're considering these different components and this is always subjective assessments. So there are no sort of rules or guidance. It's not a mechanical process in any way. So you need to make subjective assessments and therefore you also have to be very transparent with your assessments. So the first component, methodological limitations, the extent to which there are problems in the design and conduct of the primary studies supporting the review finding. And here you need to use a critical appraisal tool. As most of you know, there's not really a consensus around one of these tools. There's quite a lot of different tools out there. And the circle team are currently working on addressing whether we need a new tool for this methodology or if there's something useful out there. The second component is relevance. So here we need to address the relationship between the context in the individual studies informing them the finding and the review questions, the context for your question. So the extent to which the body of evidence from the primary studies supporting a review finding is applicable to the context specified in the review question. And there are three different types of relevance issues. We have indirect relevance, partial relevance, and uncertain relevance. And just to exemplify using the population here, so if you're interested in, for example, experiences of children 10 to 18 years, and you have studies, you're using studies with younger children, this would be an indirect relevance issue. Partial relevance could be that you have direct evidence, but only partial from the whole spectrum you're interested in. So you might be interested in all children's experiences, but you only have data from experiences of girls or asylum seekers. Uncertain relevance would be that you might be interested in these children 10 to 18 years, but it's unclear what ages the children have in the studies. So in all of these cases, you would need to consider whether you lose confidence in the review finding. The third component is coherence. So here you make an assessment of the fit between the data in the primary studies and the review finding. So you would become less confident in the review finding if some of the data contradicts the finding or some of the data is ambiguous. And I'll come back to this component when I'm exemplifying from our report, because of course, your methods for producing the findings will affect whether or not you have contradictory data. And the fourth component, adequacy of data. So here you assess the degree of richness and quantity of data supporting the review finding. You might become less confident in your finding if you have very thin data or only a few studies or a few participants. And again, this is a judgment call just like the other components. So there's no rules as to how many studies or how many participants or how rich data this unit address in relation to each particular finding. So in the end, then you make an overall assessment about these four different components. And if you have serious concerns about either of them, you go down one step in your confidence level. So this was a really quick overview about the methodology. Now I will go in and exemplify these concepts using a recent report from SBU. So just very short for those of you who don't know what SBU is. So we are a governmental agency that has done HTA, health technology assessments for 30 years. We just celebrated our 30th birthday. And we've also assessed social services the last two years. And we do not do guidelines. So our target audiences can be, for example, decision makers in health care and social services or other agencies that do produce guidelines. And the report I'm going to talk about today, we were commissioned by the Swedish government in August 2015 to assess diagnostic tests and interventions for children with fetal alcohol spectrum disorders. This is an umbrella term describing a wide range of effects that can occur if an individual has been exposed to alcohol during their mother's pregnancy. And this is a quite controversial spectrum. We realized when started looking into this. So there are people that claim that this is a quite common condition. About 5% of all children in Sweden might have the condition and it explains a lot of problems in society. There are others claiming that these don't exist whatsoever. Except I have to say fetal alcohol syndrome, which is one medical diagnosis within the spectrum, which is a quite severe condition. So we realized, okay, this is not so easy to start addressing diagnostic tests. It won't really be meaningful to start looking at sensitivity and specificity for a test if we don't even know if it exists or whether it's valuable for an individual to be diagnosed in such a way. So our overarching question in this report was whether being identified as having these conditions would improve the health and social situation for an individual or family. And we had two main questions. How did the different FASD related conditions impact the child, his or relatives and society? And what are the social, medical, economical and ethical effects of interventions for children with FASD related conditions? And as you see, these are quite big questions. So we did, I think at least four or five different systematic reviews within this report. And I'm going to talk about one of them, which had to do with the experiences of living with these conditions. So we asked, what are the experiences of living with FASD? We were interested in both individuals identified with these conditions and their parents. We found two studies addressing the experiences of individuals with FASD and 16 studies addressing the experiences of parents. And this was the team that did the synthesis of the qualitative research. So we were a group with experience in systematic reviews. A few of us had done qualitative evidence synthesis beforehand. A few of us had done primary qualitative research. And we also had expertise within the field, as well as methodological support from Heather Montecas, who's part of the coordinating team for CERQA. So before I go into the results there, I just wanted to point out what our aim was and how this affected the generation of the findings. So our aim was to highlight general experiences of these individuals. We were not interested in generating hypothesis or explanatory models. We wanted to stay quite close to the descriptive data. To generate findings that if we would sort of invite people with these conditions in Sweden and their families and present the results, they would be nodding their heads and sort of recognizing what we were saying. So with that aim, it was quite easy for us to do a synthesis based on primary studies with different analytical approaches because we went very close to the descriptive data in the studies. And we also worked with the findings until they became coherent. Which meant that some of our findings became quite general. Like we were interested in experiences of receiving a diagnosis. And there we ended up saying something like parents to individuals identified as having FASD can experience both positive and negative consequences of receiving a diagnosis. So quite wide general comment. And this was based on sort of our target audience and what we felt would be a reasonable level to address our confidence in. So this is the findings we came up with. So we had three sort of categories. We had parents' experiences regarding their child's disabilities, parents' experiences of parenthood in relation to FASD, and parents' experiences regarding society. And all of these are, so these are sort of groupings of second level themes, which we did our circle on. And I will show you that in a minute. And all of these second level themes are then supported by first level themes. So an example of one finding then would be parents' experience that living with a child with FASD burdens the whole family. As you see a quite general statement. And here you have the first level themes that inform the second level theme. And as I mentioned, one needs to be very transparent about the judgements for each of these components. So besides discussing a lot of this in the actual report, we also had tables showing all the different components for each finding. So here you can see that we ended up with having moderate confidence in the finding parents' experience that living with a child with FASD burdens the whole family. And we've rated down because we had moderate methodological limitations, as well as some minor concerns about adequacy. And we ended up summarizing our results. So this is just an example. But here we've summarized the results with the highest confidence. We had quite a lot of discussions how we go from assessing the confidence to expressing it in our report. So the process of doing this was quite straightforward for us. And we're used to doing this for quantitative data. And so on addressing how certain we are in whatever we're claiming based on our reports. But for the qualitative data, it was quite difficult to find wordings for this uncertainty. So in the end, we sort of more stated this, but based it on the findings with the highest confidence. So in the end, we sort of put all these different pieces together. We had quantitative synthesis on on prevalence of different disabilities, given that you would be diagnosed with these conditions. We found very little there. We had very little information about benefits and harms of interventions for these children. So given that there's no real point to diagnose someone, you wouldn't know what their prognosis are for different conditions and you wouldn't know what to do to help them based on the diagnosis. So sort of the most weight in this report became on the qualitative side, describing what it's like to live with these conditions. Hopefully that can inform how to address these issues in the long run. So just to summarize the circle is a systematic and transparent way of assessing how much confidence to place in an individual review finding. And you would be considering four different components, methodological limitations, relevance, coherence and adequacy of data. And the aim of this assessment will affect how the findings are generated and the confidence you will place in the findings. So in our example, we generated coherent findings, several with quite high certainty. Yeah, so thank you.