 Felly, ydych chi, Tim, ac ydych chi i'n gweithio gyrfa, oeddwn i'n meddwl i chi i'r gweithio'n gweithio'n gweithio'n gweithio. Rwy'n golygu goeth o'r cynnig o meddwlolio mewn cyllid. Felly, dydyn nhw, rydyn ni'n gweithio'n meddwlolio, ydych chi'n cynnig oedd yn ddim ein profesor o meddwlolio yn meddwlolio meddwlolio mewn cyllid. Ac mae'r ysgol arlaeddur hwnnw, rydyn ni'n golygu. Thank you very much. It's very exciting that it's at this meeting. I'm a preclinical meta research scientist. Essentially, our focus is on preclinical models of human diseases and how they translate to effective therapies and our increased understanding of human diseases in a clinical setting, so translating what we see in a lab setting into humans. I'm also a member of the Edinburgh camarades team and I'll go a bit more into camarades and I'm a black female academic. So if those of you are not familiar with camarades, this stands for the collaborative approach to meta analysis and review of animal data from experimental studies. Essentially what we do is we look systematically across the modelling of a range of different conditions. We came into this work very much focused on stroke research but this has expanded looking at a range of different conditions and it started off with us in Edinburgh and colleagues in Melbourne and Australia and also expanded to now having six national coordinating centres around the world. I used to spend a lot of time replying to emails and people saying, I'm trying to do a systematic review of animal studies, how do I do it? So then we said, OK, why don't we have teleconferences every week so people could ask questions? This has now expanded into a kind of open drop-in session every week for people who drop in. We have a retreat as well where we will try to get together and have some face-to-face time which I think is very important. One of the things that has come out of our work in camarades is this huge data repository. This repository was hosted in Microsoft Access. Anyone who knows anything about data knows that that is not fit for purpose. It had data from about 19 different diseases, 7,000 studies and over 200,000 animals. I'll go into how that's evolved away from access in a bit. But I wanted to start off with what our problem was, this translational problem, that everything seemed to work in the laboratory and didn't work in humans. So data from colleagues in Australia back in 2006 who did a review looking at all the different interventions that had ever been tested in the model of stroke, there were over 1,000. 600 in animal models, 374 of those were shown to be effective in these models. Just under 100 were tested in humans in a clinical setting and then only one of these was effective in humans. There are a couple of other interventions that are used clinically, but in stroke the data that comes from animals, when I say translated, that's also quite a loose description. There's only one intervention. So we had this problem of translational failure. With the benefit of hindsight, this is how we present our hypothesis and I'm not sure that this came later our hypothesis, but essentially we thought that there were some perverse incentives in the life sciences, like publication bias, funding promotion to produce positive results with maybe not so much attention paid to their validity. This clearly is meta science but we weren't calling it meta science at that time. There wasn't this crowd, we didn't have that language. We also identified that in the use of animal models we have all these additional pressures to reduce the number of animals that we are using because of cost, time, ethical considerations, feasibility. So then we have all these studies that are underpowered of unknown power and then all these different factors combine to compromise the utility of animal models and contribute to translational failure. So our focus was really on some of these, what are the potential sources of bias that are leading to this or contributing to this translational failure. And we had four main areas that we focused our energies on. The first being internal validity, so the strength of the cause-effect relationship, things like randomisation and blinding, the use of those to improve the internal validity of studies. We're interested in generalisability or the external validity of a study. So do the findings that we observe in a laboratory in an animal model, do they translate to other laboratories trying to do similar work to humans in a clinical setting to different time points? Construct validity, are we really measuring what we think we're measuring? Does what we see in an animal what we think is relevant to a human situation truly relevant and then publication bias? So are all the data that are being collected actually making it out into the public domain? And as I mentioned with camarades, we've done all this work through the lens really of systematic reviews, so identifying all the relevant studies for that question, distilling it, curating it into the systematic review and then using those data to answer our questions. And we found that biases were prevalent and they were important. So like I said, we started in stroke research but our work has expanded across different disease areas. We've looked at motor neuron disease, Alzheimer's, Parkinson's, EAE, which is a model of MS and glioma. And we found the reporting of things like randomisation and blinding was pretty low across the board. When we looked at the impact of this, it seems that this also has an effect on the outcomes that are observed. So these are data from one of our systematic reviews where we stratified studies according to those that do report randomisation and those that don't. And we see studies that don't report randomisation, report much larger treatment effects than those that are considered to be of increased rigor. In terms of generalisability, the standardisation fallacy is a concept that we worked with for some colleagues in burn. And essentially the efforts to increase reproducibility in studies by reducing variation by standardising everything. So things like ensuring that the lab environment is consistent all the time and that you use consistent tests and that you inbreed all these animals so that there's very little genetic variation. What it's doing essentially is increasing the risk of detecting an effect with very low external validity. Or we're missing effects where there's high external validity. And a colleague of ours, Hannah Werbal, talks about the reaction norm. So the effects that we observe are due to an interaction between our environment and the genetics. And depending on what other kind of nuisance variables are out there you will observe an effect somewhere along this reaction norm. And it could be, I don't know, the radios on or somebody's got, you know, channel number five on or something that causes a slightly different effect to be observed. But what was happening is that we see, we're doing an experiment and we see an effect here and somebody else does a similar experiment and sees an effect here and we say we fail to reproduce. But actually are we just sampling on different areas of the reaction norm. And to try and address this we did some simulation work again with our colleagues in Bern where we looked at the concept of multi-centre studies to increase heterogeneity. And if we just focus on these two extremes here, the one lab study and the two lab studies, so each of these points are individual experiments. The red line is a line of no effect. This is the overall estimate when you pull these studies together and this coloured area here is a 95% confidence intervals of that effect. We see in the single lab studies some of the experiments don't overlap with a 95% confidence interval suggesting that we've failed to reproduce an effect. But in the multi lab studies all the experiments overlap with the pulled estimate showing more consistency and reproducibility across the board. In terms of construct validity, I will go back to stroke. Animal models that we use essentially are trying to capture aspects of diseases. When we induce a stroke in a rat it's not a mini human. We're trying to recapture part of that disease model. The outcome measures that we use are often surrogates. In the stroke field, infarct size is the outcome that is generally used to see how effective your experiment was in an animal model. The brain is damaged. In humans in a clinical setting we're more interested in daily activity. Can somebody get up in the morning, get washed, eat their breakfast, that kind of thing? These are two different outcome measures. There's some data that shows these actually don't correlate that well with each other. But then trying to do the daily activities in a rat is obviously there's work in that space but it's much more complicated. In stroke time is also very important. I call this the tail of two drugs. We've got these two drugs, Tyrolazad and TPA. TPA is the intervention in the slide at the beginning where I showed the one intervention that worked clinically that had the animal data behind it. Tyrolazad is part of those 100 drugs that was tested in clinical trials and didn't work. When we went back and looked at the animal data and we looked at stroke time is very important, the animals in Tyrolazad were given the intervention about 10 minutes after they had a stroke. For TPA it was about an hour and a half later. When we look at time to treatment in the clinical trials for Tyrolazad more than three quarters of the patients were given a drug more than three hours after they had a stroke. Whereas the TPA data is more consistent with the animal data. So is it that Tyrolazad doesn't work or if you gave it to patients 10 minutes after they had a stroke maybe it would have more consistency. The final area of bias I was going to quickly focus on is publication bias. These are data from the Camaradid database, the famous access database with lots of stroke experiments. Essentially what we did is we used a funnel plot to try and look for publication bias. When the x-axis we plotted the effect size for each of the individual experiments on this plot, that's an individual experiment against the precision which is the inverse of the standard error. So the more precise the study is, the narrower the confidence interval. What we expect with no publication bias is that you should have a kind of equal distribution at low precision of positive and negative studies. As you become more precise your studies should converge around the true effect. What we observed is a lack of imprecise negative studies in our data set, which is consistent with publication bias. Then we used a technique called Trim and Fill which essentially tries to impute these theoretically missing studies so that your funnel plot is symmetrical, which are these red dots here. When we did this, our overall effect was reduced from 32% down to 26% and it estimates that about one in six experiments remain unpublished. I think this is a growth, a huge underestimation of how big this problem is. A lot of the studies in our data set, these negative studies weren't truly negative publications. There were experiments that had some negative experiments, some papers that had some negative experiments alongside some positive ones in the same paper. But it's not just systematic reviews that we do in the camarades collaboration. These are two examples of some studies that we've done. The first was in collaboration with PLOS, the Icarus study, which is a randomised control trial looking at asking authors when they submit an animal study to submit a checklist, the arrived guidelines checklist, which is a reporting checklist, which unfortunately showed that asking authors to submit a checklist doesn't lead to improved reporting. Then in collaboration with Nature, we've done an observational study where we looked at the impact of their editorial policy change requiring authors to explain how they took measures to reduce risk bias if that also improved the reporting of their studies. Our work, I think, has had some impact, some positive impact. We don't just look at people's work and tell them all the things that they've done wrong. The purpose is to try and really improve the quality of the preclinical research that's been done. This is an example from a review that we did back in 2009 with a drug called interleukin 1 receptor antagonist in models of stroke, and we looked at the reporting of these measures to reduce risk of bias, ar randomisation, blinding and sample size calculations. As you can see, this is the proportion of studies that report these items and very few studies report this. One of the main PIs in this work, a colleague in Manchester, was sitting next to him at a conference a few years later and he said to me, you know, that really annoyed us when you did that. It just really highlighted some of the issues. It was an unsolicited review of our work. We thought it was unfair because you didn't really understand the nuances. But then we stopped and we thought about it and realised if we don't tell people what we did because their argument was that they did blind some of their experiments and they had done some sample size calculations but because they hadn't reported it, their work in a not-so-good light. He said it changed how they approached their work and me being me said, okay, I'll have a look and check. We did an updated review in 2016 and we see a massive increase in the reporting of these measures to reduce risk of bias, which was a really positive and rewarding thing to see when your work is having some sort of impact. Some of the other impacts of our group has been around pushing robust design in experimental research. Working with the NC3Rs, we've had some input into the experimental design assistant, which is an online tool to support researchers in designing their experiments well. Pushing for increased clarity of how studies have been performed, i.e. through reporting guidelines. The Icarus study led to or supported the revision of the arrive guidelines so there's now arrive 2.0, which we were heavily involved in and also the NDR framework. But also push for collaborative studies. I led the multi-part group, which was a collaboration to design the framework for multi-center preclinical experiments. It was the impetus for the NIH-funded SPAN network for multi-center preclinical experiments. I keep mentioning the famous Microsoft Access Database. One of the things we were very clear about was that we need improved infrastructure to facilitate and support people doing high quality systematic reviews going forward. We developed the SURF platform, which is an online application to facilitate people doing preclinical systematic reviews. It allows the work to be done by large teams, anywhere, doesn't matter where you are. It's got facilities to help people, screenpapers independently, you're blinded to other people's annotations when you're taking data out of papers, that kind of thing. And it's not in access. And SURF is being used, which is great to see globally. We've got over 2,500 users now, 1,300 projects in the platform. So that kind of... All that talk, I guess, was trying to show you some of the work that we've done within Camerides. When I say we, it really is we. It's not just us in Edinburgh. It's colleagues from around the world who've worked with us to, I think, create some really important and impactful research. What I think some of the benefits of collaboration are, are certainly pooling resources and expertise. I think this is particularly important in an evolving discipline. I said at the beginning that, you know, meta science wasn't a term that we used in what we were doing, even though that's what it was. We didn't have the language for it. It was very new. There weren't very many preclinical systematic reviews out there. We were working with colleagues, you know, and we relied heavily on the Cochran collaboration, the clinicians who were doing systematic reviews to get their expertise. Because it wasn't really a discipline, there weren't really resources out there. There weren't dedicated pots of money to do this work. This screenshot is from our universities. So, faculty, we've got this online platform where when you're putting in grant applications, it goes through this system, and you can see that by far the majority of our applications have been unsuccessful. This actually makes things look a lot better than they are. There's lots of things that we've put in. There's letters of intent at the beginning that you spend ages doing that don't go through this because you don't have the finance bit in it, aren't in here. Although I think we appear relatively successful, it really is an uphill struggle and it has been very difficult to get, to just identifying pots of money that you can apply for and then writing those grants and actually getting the money. Learning from other disciplines, I often talk about when some of the reviews that we've done, especially in the preclinical space, you often have lots of studies. We've got systematic reviews, we've got thousands of papers in them. We used to screen these papers manually, which clearly would take a huge amount of time. I remember we had somebody who joined our team and she had a background in physics and just thought it was absolutely bonkers. She's like, why would you not just get an algorithm to do this? Why are you doing this manually? And it never even occurred to us back then that this is something that you could do. So learning from other disciplines has been transformative in how we do our research. Greater sample sizes, I think it's given, and data diversity. But also increased transparency and accountability. I think when you know that other people are going to see what you're doing, you think twice and you look and you double check your work, which I think increases the quality of what you're doing. As with all these things, they are challenges and limitations. It can be difficult coordinating large-scale projects. One thing that I've learned, I don't want to say the hard way, but it has been challenging, but different disciplines speak different languages. The same word can mean something very different between a data scientist and a biologist, or when you speak to a statistician and you think you're having the same conversation, but you're not at all. I often say, I feel like, maybe science doesn't work, I can maybe go into diplomacy now, because I feel like I've learned a lot about how to speak to people who speak different languages. Time and resource constraints. They are issues around funding across different jurisdictions. We were at dinner the other night and there was a big winch about Brexit. It's had a huge impact for us in the UK in terms of accessing... It's not just about accessing the money, it's also how your European colleagues, at least, if we have a European...a UK partner, it's a bit more difficult because now they're Brexit. It's a huge impact in terms of how collaboration works. I've spoken already about very few meta-science pots. I know for us a lot of the work that we were doing was...it wasn't funded work, it was alongside the work, the neuroscience grants that we were writing, and then on the side we were doing these other meta-research and meta-science projects. They also can be a heterogeneity of motivations. People from different areas have different reasons for what they want to do. They're motivated by different things to do this work and managing that can be challenging. Getting traction as well within a domain. I think one of the things I often talk about within the stroke and the example I gave earlier about the colleague who talked about how our review changed how he wrote his papers, getting traction within a domain is really important. It's not useful for us to, like I say, keep telling people, you're not doing it right, you're not doing it right without actually getting their buy-in and changing how they do what they do. Future directions and recommendations. I think it's a real importance in terms of diversity and inclusivity in collaborative projects. Not just in terms of expertise, I think I've spoken a lot about that. Yes, you want people from different academic backgrounds and different expertise, but I think you also want a difference in people's backgrounds. I talked about being a black female academic at the beginning. In the years throughout my career, I feel like I've often been the only black female academic in a room. There are many more women now in the room when I sit in rooms in collaborative projects, but very few women of colour or people of colour generally. I think that's something that we, as a group, need to think about. I saw that the next session actually is talking about equality and diversity, so it would be interesting to see what comes out of that discussion. Incorporating emerging technologies and methodologies. I didn't have time today to really go into some of the work that we've done to try and do that, apart from saying that we used to screen things manually and then now we use machine learning. But my colleague, Caitlin, I was saying the other day as well that this meeting is a camaraderie sandwich. I'm starting. My colleague, Caitlin, has given the very last talk of the conference, so hopefully you stay to hear that, but she's going to talk about some of the work that we do in terms of using machine learning algorithms and technologies to really step up how we do systematic reviews and things like making them living. I think there's something around developing policies and incentives to enable collaboration. I remember at the start of my career the real focus on the winner takes all mentality and the kind of individualistic approach to academia, and I think that has definitely shifted, but I think there's still a huge amount to do, and I think that is something that does differ, and something I've observed differs across different disciplines as well. So some shiny new things. I am just coming back from maternity leave and I wrote three grants when I was pregnant and all three got funded. So I'm trying to break it to my husband, but this is the key to my career is trying some grants while pregnant. I won't go into details about these different projects, but these are three projects that I'll be jumping back into when I'm back at work full-time. But what I wanted to highlight is the collaborative nature of all of them. That's really been key to what we do. It's, you know, different expertise in different countries. The IRIS collaboration is going to be presented a little bit later today with colleagues who have also had European Commission-funded projects tiered to anasaris, and we're going to talk about how even though these are big collaborative projects, how these three big collaborative projects are also going to try and collaborate with each other. And then this last one is interesting because we're also moving into this space of collaborating with people with lived experiences of the conditions that we're working in. So this project is focused on depression, anxiety, and psychosis, and we've got representatives of people with these conditions in the project, which again I think is really important to ensure the work that we're doing is useful. I think what's helped has been finding our tribe. Metascience I think is now a discipline that's all right. But before that, finding other people who had those interests I think has been really important. Finding your cheerleaders, and for us those have come from different stakeholders. Working with people like PLOS and nature I think has been really important for us. Face-to-face time, really taking time to develop relationships I think is essential to effective collaborations. I mentioned we have a camaraderie's retreat. I think this is it works very well. Within that retreat we also have a mentorship scheme to ensure that junior researchers are mentored and are supported in their careers. So, this is my last slide in the summary. We didn't know that we were doing metascience at the beginning. We showed at least in our discipline that many studies were at risk of bias and bias studies over inflate effect size. I think we've had some impact on which has been rewarding to see. But I think the underlying the thing that I want to underline here is that collaboration has really been key to what we've done. So thank you all for listening and to everyone who supported our work. So I think we have time for just maybe one, two questions if anyone wants to go up there. Presentation and you mentioned bias and random error. How about our cognitive bias? How to tackle the psychological aspect of a scientist wanting positive results and not not exactly searching for the truth but for positive results and publications. So how about our cognitive bias? Which is natural of the human being? It is very natural and I think that's part of that is you're right. It's natural. I think it's interesting. It's exciting. You want to see positive you want to show positive things. You want things to work. In terms of how you tackle that that's definitely beyond my expertise. I think I would have to guess the incentives I think will make a big difference. How you incentivise this stuff I think would make a huge difference. Because as much as we want to just do the things that we like doing you also kind of have to put your big boy pants on essentially and do the things that you don't like doing because they're important and essential. But how you go about that, I think that's definitely beyond my expertise. Manu Baker from Issues in Science and Technology. I know there's been some efforts to have animal registries for pre-registration specifically for animal studies and I'm just wondering how feasible you think that is and how useful it will be to see some of the unpublished work. So they do exist actually I'm aware of two in Germany there's the BFR have one and there's the preclinicaltrials.eu as well. I think part of it's a cultural issue in the we animal scientists haven't traditionally pre-registered their studies but they are I think it's one of these movements isn't it to see adopters I think there's a thing around incentivising it some of the arguments around that have been the nature of some of the studies you know if you're doing a confirmatory experiment and you've got a very clear hypothesis driven study then it's very you can see how you would pre-register that I think some of this kind of blue sky exploratory stuff people are a bit more concerned about how you would do that and how it would work so I think there might be some kinks that iron out but I think it's feasible and I think it would have an impact in terms of being aware of what's been done and it would also sign kind of to the question before around being aware that these experiments have been done without necessarily having to then write a full paper about something that you maybe hasn't hasn't worked not because it hasn't worked because it's you haven't got the result you wanted but it hasn't worked because for technical issues or for you know some other reasons so I think there is there's a supporter of pre-registering animal studies for sure and I think there is potential for impact when we're on to the plan