 Hi, everyone. Thanks so much for having me speak at this forum and today I'm going to talk about open scholarship and The subtitle of my talk is where are the self-correcting mechanisms of science and I'm going to argue that openness and transparency Are fundamental to making science live up to its reputation for valuing self-correction So we know from research in the US the Pew Survey that public trust in science is consistently very high in this these results the public trust in Scientists are the green bars and you can see from 2016 through 2019 The public trust in scientists is quite high around 80% of Americans say that they trust scientists At least somewhat a fair amount or a great deal and that's higher than public trust in most other institutions So we know that Americans trust science and trust scientists quite a bit But another interesting thing in the Pew survey is that Americans are not naive about what scientists are like When the Pew researchers asked Americans about their views about research scientists in medicine nutrition and environmental science And they asked specifically do you think that science research scientists in these domains Provide fair and accurate information or transparent about conflicts of interest Or my favorite admit and take responsibility for mistakes The proportion of respondents who say yes is actually quite low. So that's the green dots in these results So around 10 to 15 percent of Americans say that research scientists in these domains admit and take responsibility for mistakes So why do they trust scientists if they don't they aren't deluded that scientists are these noble creatures that always tell the truth and Are always accurate and fair and take responsibility for their mistakes Well, one answer to that question of why do people trust science? Is that the trust in science doesn't come from individuals individual scientists being noble and don Campbell a psychologist said this well He said the resulting dependability of reports comes from a social process rather than from dependence upon the honesty and competence of any single experimenter He said organized distrust produces trustworthy reports So it's this social phenomenon this community that values checking each other and that doesn't just Automatically trust each other but actually has this organized skepticism or organized distrust of each other And that's what makes science trustworthy And actually this phrase organized distrust comes from a sociologist of science Robert Merton who wrote in the 1940s And he talked about the core values or norms in science And he said that there are four values that kind of define what it means to be a scientist And he was talking about this social process what what makes a community of scientists bind together It's their shared values and specifically he thought that there were these four values at the center of it So i'm going to go through them very quickly The first one is the idea of universalism Which is the idea that the validity of a scientific claim doesn't depend on who's making it It doesn't matter if it's a harvard scientist or a graduate student or somebody from another country We evaluate the validity of the claim on its own merits So status shouldn't matter a hierarchy shouldn't matter This is obviously important for self-correction because if we don't have this principle and this value Then the most elite most eminent scientists will never get corrected when they make mistakes The second value that merton talked about was communality or he called it communism And this is the closest to the core of open science It's the idea that the findings of science belong to everyone. They're not private property So open communication is key. There should not be secrecy And of course you can also see why this is important to self-correction, right? If we don't put things out in the open if we don't show our work, then we're not going to find the errors And i'll come back to this idea The third value talk value that merton talked about was disinterestedness And this is the idea that scientists should be focused on finding the truth not on their own success So self-interest shouldn't factor in you should report whatever you find even if it makes you look bad This is probably the most controversial of his norms and many philosophers and sociologists and historians of science actually argue that we don't need individual scientists to be disinterested But I think one thing that most scientists do agree on is that the self-interest of the individual scientist is not what's important to the community So if Criticizing a scientist means that you might hurt their financial gains. That's okay. We're not going to prioritize their own self-interest We're going to prioritize finding the truth at least at the community level And then the fourth norm and you can see why that's also important for self-correction Because sometimes correcting things is going to hurt people's reputations or financial opportunities and so on The fourth norm is organized skepticism or organized distrust And this is the idea that we don't take things at face value and we don't ask other scientists to take our claims at face value We expect others to want to verify our claims. That's normal. It's acceptable Um, and nothing is sacred. So no matter what I say no matter how much I want to believe someone's claim Or it lines with my moral or political or religious values. I'm still allowed to challenge it and that's still okay There's nothing that's beyond criticism or verification And obviously that's also very important for identifying mistakes and fixing them um a relatively recent survey 2007 of um Scientists asks ask these scientists. Do you subscribe to these norms? So they describe them in plainer language like the idea of sharing everything for communalism Or the idea that claims should be evaluated on their own merits for universalism um And the top two bars show the great part of the bar is the proportion of scientists who endorse the the norms The black part is the proportion of scientists who endorse the counter norms. So secrecy hierarchy um, and then some were unsure in the middle um, and um The next two bars is now asking not what do you value? But how do you actually behave and still most scientists say yeah, my behavior is in line with the norms Not many scientists are saying their behavior is in line with the counter norms But the third set of bars the bottom two bars is asking now. What about other scientists in your field? How do they behave and now we see a very different picture people say that the typical scientist Does not behave in accordance with the norms and in fact much more frequently behaves in accordance with the counter norms so So this raises the question of is science really are the scientific communities really committed to these norms and are If not then are they really self-correcting because we talked about how these norms are so important for a culture of self-correction um But these norms and the idea of these self-correcting mechanisms get raised a lot and just on twitter in the last few weeks Um, I encourage you to search on twitter for the phrase self-correcting mechanisms or self-correction and science It's it's quite fun. Um, I've blacked out the identity of this person because I didn't ask for their permission to share their tweet But it's an editor-in-chief of a major journal high impact factor journal It was in a conversation about a problematic paper at another journal and people were debating whether that paper should be retracted And they said retraction should be reserved for publication misconduct Otherwise, let the self-correcting mechanisms of science take their course And we often hear this and I used to teach this in my undergraduate research methods class I used to teach that science is self-correcting And often when we talk about self-correction in science It's as if it happens magically like if we just wait things will self-correct And this was captured really well in a tweet by james heathers A few years back where he said science is self-correcting. Sure when we correct it not because of magical progress And so what I've been asking myself is what do these self-correcting mechanisms look like? How do we know if a scientific community really is committed to self-correction? And I think that's at the heart of what I've called the credibility revolution a phrase I borrowed from other researchers Which is another kind of word for the replication crisis that's happened in my field of psychology And I think what's at the heart of the credibility revolution is trying to change our norms and practices so that we actually carry out These self-correcting mechanisms that we preach about that we say are the reasons why we should trust science And I think it really is at the core of why the public trust science Americans don't trust scientists because they think we're individually noble But I think their trust in science has something to do with the idea that we're committed to as a community Checking each other and correcting our mistakes So what are the pillars of self-correction and of credibility? I'm going to argue that there are two one is transparency and open science and that encompasses a lot of different things So things like open access to scientific outputs open access to the data and code underlying those results Openness and transparency in the peer review process and many other things It also an important pillar of openness transparency is a level playing field making sure there are no barriers to entry Making sure that we're inclusive so that Everybody can participate But there's another pillar To the credibility revolution because transparency is necessary But it's not sufficient for self-correction. So let me illustrate why So if we have more transparency that does increase the chances that there will be Identifying mistakes and correcting mistakes and that we will earn more credibility That's certainly one path that we could go if we are more transparent We show all our work people are like, oh, that's good But over here you made a mistake and we fix it and people trust us more But another possibility if we start showing all our work Is that people will find many many mistakes and mistakes that should not have been made unforced errors And that will actually lose credibility So transparency doesn't guarantee credibility transparency just guarantees the credibility you deserve And the difference between these two paths is just the difference in the quality of your methods the quality of your work So transparency is one necessary component for credibility, but it's not enough You also need to be doing good work because if you transparently do poor quality work, you will not earn more credibility This was articulated really well by Andrew Gelman In a paper he wrote called honesty and transparency are not enough and I'll just share a quote from that paper He wrote consider the practical consequences for a researcher who eagerly accepts the message of ethical and practical values of sharing and openness But does not learn about the importance of data quality He goes on he or she could then just be driving very carefully and very efficiently into a brick wall Conducting transparent experiment after transparent experiment and continuing to produce and publish noise So transparency is necessary, but it's not enough for credibility for becoming a self-correcting science The second part the second pillar i'm going to argue is something i'm calling quality control It just means having high standards making sure you're doing good solid rigorous research So this is closest to the norm of organized skepticism that mern talked about Um, and it's the idea that we should be going in and checking each other's work We shouldn't ask people to take our work at face value And it should be okay to question everything So one metaphor i've used for these two pillars of openness and quality control Is the metaphor of checking out a car a used car before you buy it So open science is saying look as a seller you have to lift the the lid you have to show what's under the hood Asking people to trust your scientific claims without showing them the underlying data and code and methods and so on Is like asking them to buy your used car without looking under the hood. That's not a reasonable request That's not what it means to be engaging in science So we have to lift the hood look under make it possible for people to look under One metaphor I like for this is giving your critics ammunition You should be giving your critics everything they need To find the errors in your work to find what's wrong with your work and fix it But the second part of that is that we need people to actually come in and look under the hood And look for errors So it's not enough to make the stuff available to give your critics ammunition There also needs to be a community that rewards and incentivizes people going in and checking and verifying and correcting So both of those are really important and I think we talk a lot about open science and more and more and that's great And i'm glad we're making a lot of progress But i'm going to argue that the second part of that also needs a lot of attention and that's the actually doing the criticism and correction So now i'll talk a little bit about the empirical evidence on how we're doing on different indicators of these two pillars of self-correction openness and transparency and quality control so My the idea behind this part of the talk is that we shouldn't be just be talking about these values. We need to actually Implement them. We need to do them in a way that's measurable and visible And then meta scientists need to track them need to say how much does this scientific community engage in practices that enable self-correction And track and compare over time compare across scientific communities and so on so that we have a way To evaluate scientific disciplines and communities and say which ones are more credible more committed to self-correction than others Which ones are making progress? um and so on Okay, so on the transparency side of things some indicators of a commitment to transparency Are the frequency with which researchers share their data and code Share their materials lab notebooks procedures, etc Pre-registers, so that's a form of transparency because pre-registration is writing down what you plan to do ahead of time So you're letting other people see the order in which you made your decisions the timing of those decisions Instead of just saying trust me. I made this decision a priori. It was planned and so on so those three have been really Visible changes that have happened in psychology And some other fields as well. So i'm going to show some data mostly from psychology, but some across social sciences on how How much we can quantify the change in these practices over the last decade or so So first on open data, so it's for a long time now. It's been Standard that when you publish in many psychology journals as an author You sign something saying that you promise to share your data with other researchers who seek to verify your results So some researchers all the way back in 2005 They contacted authors who had published papers in 2004 in journals that had this This requirement that authors promised to share their data with other researchers for verification purposes And asked for their data for verification purposes And actually their project was meant to look at a question downstream from that They wanted to get the data and then do something else with it But they ended up publishing a paper Just about the experience of trying to get the data from researchers who had published in a journal that required them to share the data Upon request and what they found was that the vast majority did not share the data for various reasons. So For 11% share the data after the first request and another 16% share the data after reminders So overall a quarter or a little bit more than a quarter of the papers. The data were made available 35% refused to share the data 14% just didn't reply And 20% promised the data but never delivered Okay, so this was before the replication crisis and the credibility revolution How have things changed? So these results are from a survey of social science scientists So it's across several different disciplines not just psychology and these are self reports Asking scientists to retrospectively report. What was the first year that they remember engaging in various open science practices? So the bottom line is pre-registration. So you can see Fewer than 20% of the social scientists surveyed here. And these were people who got their phd's prior to 2009 About 20% in the latest year 2017 Were had pre-registered at least once and that was a fairly recent thing for most of those researchers But more of them 44 percent by 2017 had posted their methods materials study instruments And 73 percent had posted their data and you can see the trajectory at least according to their own memory of when they started doing that and 84 percent had done one at least one of these practices So we can see that this period of the last 15 years or so was one of a lot of growth among social scientists for these open practices Another kind of openness I mentioned is pre-registration. So this is Writing down in a timestamped document that you make public later on What your plan is so that people can see if you deviated from your plan Which is important for interpreting the statistical results to make sure that it's not just a post hoc story. You're telling about an unexpected finding Um, which is okay to do but not to present it as if it was planned So letting the reader check what was planned and wasn't is really important for letting the reader decide for themselves If they think the result is really solid So, um, this is a graph showing what happened at a particular journal psychological science After they implemented just a simple nudge to try to get people to engage in this practice Actually, no, this is not for pre-registration. I apologize. We're going to get to pre-registration in a second This is for open data So this journal psychological science actually offered badges for open data as well as pre-registration and open materials But this graph just shows a proportion of authors who shared their data Um Before and after this policy went into effect at the journal So this journal isn't is the black line the red dotted line separates before versus after this badge policy went into effect And the other gray lines great outlines are other journals over the same period of time for comparison point in the similar in the same field psychology journals So the y axis only goes up to 40 percent here But you can see that already within a couple of years of the badge going into effect The proportion of authors sharing their data went from less than 10 percent to about 40 percent What about pre-registration so in a different Survey looking at pre-registrations on the open science framework um, the researchers found that the number of pre-registrations cumulative number of pre-registrations from 2012 2018 went from based from zero to over 18 000 pre-registrations by 2018 And a similar trend um on a smaller scale is happening with journals offering something called registered reports So registered reports is where you not only pre-register your plan, but you actually submit your paper With its pre-registration plan for peer review before you run the study So you plan your study you describe what you're going to do you submit it to a journal The journal sends that out for peer review if it gets accepted The journal is committing to publishing the paper regardless of the results as long as you seek to the plan Um, so the number of journals offering our registered reports has gone up quite a bit was 120 in 2018 I think it's over 200 now. I'm not positive, but It has continued to grow So we're doing pretty well on sharing data sharing materials pre-registration certainly the trends are fast change on those dimensions What we know less about or at least I couldn't find very much evidence about is trends in terms of open access or posting pre-prints Um open review. So making the peer review process more transparent This is one that's near and dear to my heart as a journal editor and as an author I feel that we really need to bring a lot more transparency and accountability to the peer review process We let editors get away with too much um Removing the barriers to entry and making sure that there's a level playing field so that people from different backgrounds different Demographics different geographical regions different languages etc Are able to participate fully in the scientific process And declaring conflicts of interest some fields do much better than others psychology is terrible on this front And another kind of transparency is transparency about who did what so moving away from an authorship model Which you earn authorship by being part of the writing But science doesn't really work that way a lot of people contribute important parts to the project That are different than the writing and so we need a model Not where not only we let those people be recognized for their work But also we name who did what who's responsible for what and provide more transparency about the process in that way So we need more meta science to track not only these Things that we already are tracking like open data and pre-registration But also many other aspects of openness and transparency that are vital to credibility and self-correction So moving on to the other kinds of indicators of Credibility and a commitment to self-correction quality control So here One obvious kind of quality control is detecting errors And this is tough, but some kinds of error detection are really really easy to do So some researchers developed a tool to check for one very simple kind of error It's just whether the statistics reported in the paper are internally consistent Whether there's any contradiction in the statistical reports themselves So not whether they're correct. We don't know whether they're correct or not, but just they don't have an internal contradiction Um, so these researchers developed this tool called stat check Which can automatically scrape and extract statistical test results that are reported as long as they're reported in the apa style the psychology format And so they scraped a bunch of journals eight psychology journals from 1985 2013 And they looked at what proportion of articles had at least one inconsistency So this would be like a t value that doesn't match the degrees of freedom and p value um And they found that almost 50 percent of articles had at least one inconsistency somewhere And then they looked okay, but how many of those actually change whether the result is significant or not So it was reported as p less than 0.05 But if you recalculate the t value and the degrees of freedom you actually get a non-significant result or vice versa And they found that 13 percent Had a gross inconsistency as an inconsistency that affected whether the test statistic became significant or not These are what I call unforced errors This is something that you just need to double or triple check to avoid making this error And the gross inconsistency is one that you really need to get right, especially given how much we valorize statistical significance Which is a problem for another day Um, so there are more and more tools like this being developed to detect these easy to detect errors these low hanging fruit implausible results that just are Um inconsistent with other facts within the same paper, which is just the easy stuff to detect There's a lot more kinds of errors that are harder to detect Um, but there's a lot more work to be done in error detection And some people are really rising to this challenge of wanting to give their critics ammunition and wanting to find out where the errors are In their paper one very recent example of this is Nicholas coals who literally paid other researchers to find errors in his work So this is called the red team approach So he put three thousand dollars towards incentivizing people to find errors in his work This was a project he'd already completed. He'd written up the manuscript before submitting it to a journal He asked a team of of critical scientists to try to find errors in his work Now ideally we would find the errors before the study is run So some of in this case some of the errors that were found were the designs Design problems with the study which he couldn't easily go and fix. He would have to run a new study So this is where the registered report model really shines It provides authors feedback And an error detection before they run the study so that they have time to fix Aspects of the design that can be improved. So registered reports as I mentioned before is where the authors Developed their idea designed their study Write up their paper about what they plan to do Submit it to a journal and get peer reviewed and then collect and analyze the data write up the results And they get published after that So this is a really proactive kind of prophylactic approach to error detection detecting the errors before You collect the data so that you have time to fix them There's also tools for bias detection Which is where you could take a set of published literature and ask does this look like an unbiased set of results? So meta analysis is supposed to do that We've recently learned that are well, some people knew it for a long time, but it's become more Well known that many popular meta analog techniques actually don't control for bias in the literature And one of the big new ways of thinking about and detecting bias has to do with the distribution of p values In the literature and I don't have time to go into detail But the idea is that when findings are unbiased when they're true positive results The curve of the p values the shape of the distribution of p values in that literature should be extremely right skewed Most p values should be very close to zero and very few p values should be around 0.03 0.04 or 0.05 0.06 So we can one way to get at how much bias there is in literature is looking at how much the curve of p values does not match This unbiased curve Another important aspect of an efficient quality control system a scientific community that really wants to catch errors and fix them Is the diversity of that community? It's really important that that community include people from different backgrounds with different viewpoints different values Different experiences and who will detect different kinds of problems and biases And this was captured. Well, um in Naomi Orescu's book why trust science? Especially in the part where she was describing the philosopher of science Helen Longinows approach to to her view of how to make science robust Um, and so she's describing Helen Longinows work And she says the greater the diversity and openness of a community and the stronger its protocols for supporting free and open debate The greater the degree of objectivity it may be able to achieve as individual biases and background assumptions are outed As it were by the community So the idea is that if everyone has the same assumptions coming in we're not going to catch each other's mistakes So you really want a diverse group? We're going to come at the the research with different assumptions different biases different blind spots So this is a dimension that we can also evaluate a field on how is it doing in terms of making sure people with these different backgrounds assumptions and biases are part of the community Another form of quality control is testing whether the work that's published and that we rely on is reproducible And I'm using reproducible here to mean if I use the exact same data that the authors collected and analyze it again Do I get the exact same results? So just one example of a test of reproducibility Tom hard work and his colleagues took the opportunity when a journal changed its policy to require authors to post their data So not just to say make it available upon request, but actually post your data with along with the paper So the data were now all there with all published papers So tom hard work and his colleagues went in and took 38 papers published in that journal after they required Data posting and tried to reproduce the analyses in that paper. It took a really really long time He gives a great talk about this project I just have one summary slide from the project So they categorize the papers into whether they were able to reproduce all of the results easily without assistance from the original authors Were able to reproduce all the results after getting assistance from the original authors or were not able to reproduce all of the results even after assistance from the original authors And this is what happens and this is a journal again I'll remind you these were authors who knew that they would have to post their data They did post their data So presumably if anything they were more careful to make sure their data Could produce the results in their paper But only about a third of them were reproducible without author assistance Another third roughly a little less were reproducible with assistance from the authors and then A little more than a third were not fully reproducible despite author assistance So again, I think this the results for this particular journal in this particular set of studies is actually probably too rosy because when authors Know they don't have to post their data. They might put less work into making sure that their results are reproducible So we have a lot of room for improvement here What about replicability? So this is where we don't use the authors data. We go and collect our own data We replicate the experiment anew click new data analyze it see if we can get the same result that the original authors got or a similar one I don't have time to go into this and brian will give a talk entirely on replicability So i'm just going to very quickly summarize the results of a number of replication projects where The authors went out and replicated original studies In the social sciences and these are all projects that had quite high standards for the replications much larger sample size Almost in almost all cases the replication had a much larger sample size than the original was pre-registered There's a lot of careful precautions taken And despite that only 92 out of 199 of the effects were successfully replicated So it's a 46 application rate which means 54 and I would put plus or minus 15 percent or something around that estimate Might be false positives So this also leaves a lot of room, right? This is not ideal for our top journals Many of these are replications of studies published in top journals So we have a lot of room for improvement But this is the kind of data that meta science can produce to tell us Okay, are we doing okay on this aspect of quality control? Another important aspect of a community that values self correction And quality control is making sure that there's a way for negative results to get out there Whether those are replication studies that failed or just a new idea That was a hot topic and it was tested and it produced an all result So we know from several different studies and i'll i'll skip the details But we know from several different studies that many sciences publish very very few null results It's not uncommon for journals to be just full of 80 90 positive significant results And that's definitely true for psychology Um, I won't explain this graph, but basically over 90 of articles published in psychology journals report positive or significant results That can't be accurate There's just no way that we're right 90 plus percent of the time and our studies are so high power that we always detect That true significant effect that's there So i'm going to show you next is what happens when we remove a lot of the bias that leads to this Overabundance of positive results And that's what registered reports can do because remember with registered reports the journal agrees to publish the result No matter how it comes out. They approve the protocol. They like the idea. They like the design of the study They think it's rigorous. They think we'll learn something no matter what happens So the journal promises to publish it regardless of what happens So that removes a lot of the bias on the part of the journal And the authors also have to follow the protocol which includes an analysis plan and so on So there's not a lot of room for the authors to eke out a significant result either So the bias on both sides is greatly reduced So what proportion of registered report results are significant results? So a couple of different projects have estimated this based on the now the small but growing number of registered reports that have been published So first we see that in the general literature something like 5 to 20 percent of results are significant That's what I just showed you before in psychology It's about 5 or 10 percent of traditional articles not registered reports But just regular articles have null results very rare Among registered reports. It's 55 to 65 percent of the results are null results and about 55 percent for new studies and 65 66 percent for replication studies Another project looking at a slightly different set of registered reports came to a similar conclusion So first they looked at the traditional literature again found over 90 percent of those Claimed to have found significant results And then in the registered reports literature where bias is greatly reduced It was less than 50 percent had significant results So again It's a healthy science is one that allows these negative results to get out there It doesn't hide them away or force the authors or the the the science to turn them into positive newsworthy flashy results And then another aspect of a healthy science is to have mechanisms for post publication peer review So this means that even after something has a seal of approval of the top journal Even after it's made the rounds in the news etc. There is an opportunity to correct it if it's wrong And this is really really tough There's really not much glory in this as Kurt Vonnegut said another flaw in the human character is that everybody wants to build and nobody wants to do maintenance And I think psychology in the last 10 years has shown that not only is there very little incentive and appetite for doing this work But we actually impose a cost and a disincentive on doing it And here are just a few of the names that people doing this kind of work have been called in psychology These are Well documented insults that have been slung although actually accuracy fetishists. I'm not sure where it came from. That's one of my favorites But there's a lot of Downside to being one of the people that does post publication review that criticizes published work That does replications and so on So we really need to change that if we're going to find errors Then we need to incentivize and reward and thank and be grateful for the people who actually find and correct those errors Um, and in fact, there's a lot of cases where that's our job is to evaluate work after it's been published When we hire people when we promote people we have to decide if their body of work That's already been through peer review is solid high quality work that represents the kind of work We want to hire into our departments and the Sephora the declaration on research assessment is about Making sure that the evaluations in those contexts in those high stakes contexts Are not based on bad metrics like journal impact factor But actually on the quality of the work. So that means we need a way To identify flaws even in published work We're often in a situation where we have to decide is this work that's been through peer review Is it really good high quality work? My own university university of Melbourne has recently signed Dora. So the question is what do we do? How do we do a post publication evaluation of work identify how solid it is identify flaws And Dora itself tweeted recently that one of the main three main themes that runs through Dora is the need to assess research on its Own merits But there's very little guideline on how to do this and as I mentioned very little reward and in fact a lot of punishment For actually trying to scrutinize and assess published work on its own merits Thanks to some funding from fetzer. My graduate students and I are working on a rubric to try to evaluate flaws in published work So because we work in social and personality psychology, we think about things like construct validity Statistical validity internal validity and external validity. So we're working on this rubric Sarah Schiavoni is the main person behind this And we're trying to validate the rubric and see if this helps structure post publication peer review and provide quantitative metrics of the quality of published work So I've talked about openness transparency and quality control as two of the pillars of credibility and self-correction And I also think that's the answer to the question. Where are these mysterious self-correcting mechanisms in science that we keep talking about? They're not magic. This is what they look like. They look like openness open data Open access to publications openness in the review process They look like detecting errors correcting them making sure things are reproducible replicable making sure there's diversity of voices In the criticism process publishing null results, etc So in her book white trust science Naomi Reska says that when we observe scientists We find that they have developed a variety of practices for vetting knowledge for identifying problems in their theories and experiments and attempting to correct them And that sounds really nice and I hope that's true But the truth is we don't actually know and I would argue that as scientists We need to stop telling the public trust us we have these self-correcting mechanisms in place And we need to start making them visible need to start being able to point to and quantify To what extent do we have these self-correcting mechanisms in place? And in fact james heathers who's done a lot of this correction himself Um has said in reality the mechanisms to correct bad science are slow unreliably enforced Capricious run with only the barest nod towards formal policy confer no reward and sometimes punitive elements for a complainant who might use them It's time to start actually measuring and looking for these self-correcting mechanisms And I think we need to look for them in both categories of transparency and openness and quality control. Thank you