 Hi, everyone. Welcome to this lightning talk session. My name is Zach Lumis. I am a project coordinator at the Center for open science and your moderator for this session. A few housekeeping or logistical notes before we get started. Each presenter will have eight minutes to present. And I'll keeping time and so presenters reminder I will step in if you start to exceed your time. And during those talks, please feel free as the audience to submit any questions to the Q&A. We don't have any dedicated time to answer those live but the speakers will be able to answer them via text. And if we have any time at the end we may try to fit some in live. So without further ado, I will get us started. The first talk is going to be by Nick Fox, giving a review of the metascience 2019 lottery grant. Just setting up here. Can you all see that okay? Zach, you can see that okay? Yep. Let's see, like that. Okay, great. So thank you very much everyone for coming. I hope you've been enjoying the conference so far. I have definitely. My name is Nick. I'm a research scientist at the Center for Open Science. I just want to give a really quick review of a really cool kind of pilot program that kind of stemmed out of the first metascience conference metascience 2019. Let's see if I can. There we go. Okay, so background. Metascience 2019 was a conference in 2019. And there, Carl Bergstrom, who's a professor at the University of Washington, gave a really interesting talk, the videos online, you can go watch it. On the inherent inefficiency of the grant proposal kind of system as we know it, the competition to award and win grants. And in that talk, he modeled how public and private research interests, so both advancing science as well as advancing personal careers, coupled with the grant review process and kind of the inefficiency or, you know, lack of perfect ability of reviewers to assign quality to grant proposals, how it affects research output. And what he found was the low paylines that are involved in many grants, coupled with the high cost of writing competitive grants, can actually generate a negative return on investment where it actually be a better time for say the NIH or another grant offering institution to just put the money in a bin and burn it. That's his words not mine. So if you don't believe me here's a very meta conference for you. This is white man, giving a conference talk about a white man giving a conference talk. And here's just one of the slides about the benefits of a potential lottery system so instead of doing current grant writing system as we know it, what if there was a lottery involved in how scientists received the grant funding. The talk is really great. So the feds are Franklin fund, who was one of the funders of the medicine is 2019 conference said you know, but maybe there's something to this can we can we do a pilot kind of study this and look at it. And so, with their support, there was a modified lottery pilot study done and there's a link to the summary, be able to share these slides hopefully. The background the grant proposals were one page, and they were capped at $20,000 budget so the budget range was five to $20,000. The authors and their institutions were blinded from their review pool. The emphasis on those grants were on innovation and not likelihood of success necessarily. And so we had to review peer process where all proposals that met the funding threshold were put into a lottery, and really it was, is the proposed work feasible, given the constraints of time and budget. So there were 90 proposals that were received and 71 of those met the threshold and were entered into the lottery, and from those 71 32 proposals were funded. And as part of this there were three survey waves to kind of better understand people's beliefs and opinions about kind of grant process in general, as well as their feelings towards the field of medicines. So, we asked before awards were given after awards were given and a year after the words were given to all people who submitted so the 90 people who submitted about what percent of their research time in the last year has been devoted to and while there is changes in both directions are small for those that didn't meet the funding threshold there was a small decline in the amount of time spent working on medicines projects same with those who entered the lottery but were not awarded. Though for those who were awarded in the lottery. There was slight increase or at least sustained interest in doing medicines over that 12 month period, which is interesting that these small dollar grants may propose may sustain interest. When we ask about what their beliefs are in terms of how much time should be spent working towards receiving grant funds. This is kind of a match and I'll try to walk through this. So the top row is how much time do you believe should be spent in applying for funding. The middle row is how much time do you believe should be spent conducting research and the bottom row is how much time do you believe should be in writing up research reports. The first two columns is looking at personal use of time. So before the award participants said they spent about 14% of their time in applying for funding and one year post it's about 12%. And that breaks down to 53 5632 so it's kind of stable across a year. Their personal preference would be spending about 8% of their time applying for funding so kind of half or a little more than half with over 60% of their time conducting their research and about 30% of their time writing it up. And then we asked about what what do you believe the kind of preference through the field is it matches pretty much the same 8% applying to do research about 65% conducting the research about 2526% in writing the research. And this is kind of my final data slide. So, when we asked participants you know how how should one go about kind of reviewing the merit and goodness of proposals. There were kind of a few different relationships I just want to point out. So grant allocation based on for the proposed work was the merit of the work being proposed with or without lottery was preferred to hear positive values are more favorable zeros neutral negative values less favorable grant allocation based on the Proposer. So just on the merit of the proposer that's a 10 year faculty or a graduate student or what have you. That was seen as pretty neutral. A purely blind lottery, where it was just random blind to the proposer and to the work was not seen favorably. And when we compare short proposals to long shorter proposal was preferred to a long proposal. The project scope was supposed to be a year, but that that conference was in the end of 2019. And as we all know coven 19 has happened and is happening. So these timelines have been extended to the projects that have been funded are currently still ongoing. So the main takeaways is that there's continued interest in partial lottery programs like the one piloted participants continue to believe that the current system of grantsmanship takes an increased amount of time away from doing research and writing and also this is also a model in more detailed Carl's talk. And the merit of proposed work should be considered for awards, not solely the proposer, the mix of both the merit of the work and the proposer was seen as as slightly favorable as well. And that's it. Thank you so much. If you have any questions drop me in the Q amp a and I look forward to talking all later. Thanks. Thank you, Nick. While you wrap up, I will introduce our next speaker who is Dominic Russia, who will be presenting the quality of open data sets shared by researchers in ecology and evolution is moderately repeatable and slow to improve. Dominic the floor is yours. Great. Thanks very much. Let me just try to share this. Is that working can everyone see my screen. Great. So thanks a lot. And thanks everyone for attending the session. So I've actually had to change the title of my talk, because it was too long for the interest slide that we were given. So this title is a bit bolder than the original one, but I think I think given the data that I have to show today. In fact, I'm able to paint a fairly good picture about the state of open data in ecology and evolution. Okay, so let me start by saying that in biology and probably also in lots of other disciplines. There's pretty solid evidence so far that mandatory open data policies have had a really positive effect on data sharing. So essentially when journals require researchers to share data, they deposit more data into repositories. And this is obviously a good thing. The important question is, does more data also mean better data. And that's particularly important I think because we know that most journals and also data repositories don't necessarily have the resources and sometimes also the will to look at the check the quality of the data that's being shared by researchers. So we're looking into this question with some colleagues back in 2015. And we want to figure out what is the quality of the data that's being shared by people in our field. So in ecology and evolution. And bear in mind that this was done before the fair sharing principles came about. So this study is a bit dated now but it's important I think because it sets the stage for the new data that I want to present today. So bear with me for just a minute, while I explain the methods that we used back then. So to do this study, we assessed 100 non molecular studies in ecology and evolution that were published in journals that have a mandatory open data policy. We looked at 50 studies in 2012 and 50 studies in 2013 that were published in the seven journals. And all of the data sets that we looked at were shared in the repository dry out which is a very common repository in my field. So we assess the quality of these data based on two different criteria. The first one was completeness. So are all the data necessary to reproduce the results in theory available in the archive data set. That's what's required by the journals policy. But we also looked at reusability, which is something that's not required by journals but obviously still very important. And this has to do with whether there's metadata to understand the data set, whether the data are shared in a format that facilitates reuse. So, I'll show you how we kind of coded these things you don't have to worry about the details here but we essentially developed a scheme basically a scoring scheme to assess completeness and reusability. So here a score of four or five meant that the data sets complied with the journals open data policy whereas a score of three or less meant that it failed to comply with the journals policy. And the same thing for data reusability, except that, as I said it's not a requirement so we basically gave the studies a pass for good reusability practices versus fail for reusability. And so what did we find at the time for these studies in 2012 and 2013. Our results showed that over half of the studies didn't meet the minimum requirement of their journals open data policy. And the results for data reusability were even more stark. We found that 64%. So almost two thirds of the studies were archived in a way that either partially or entirely prevented the reuse of the data. So now this year, I decided it was a good time to check whether anything had changed and whether the quality of the data that we're sharing in ecology and evolution is improving over time. So we use the exact same methods I just described, and we check 25 studies per year again in journals that require data sharing as a condition of publication. And what we found isn't really what we had hoped for. There is absolutely no change in both the completeness and the reusability of these data sets over time. So clearly, these data are a bit depressing, but they're also very interesting, I think, because even though the trends are really flat, there's a ton of variation around in the data. So some data sets are pretty average. Some are really bad, but a lot of them are also very good. And then a logical question to ask when you see this these data is what what explains this variation. And to answer that, I teamed up with some students and some colleagues here in Canada, and we collected a different data set. We did that by identifying all of the profs in ecology and evolution in Canada's top 20 ranked universities. There were 351 of them. And then we checked what percentage of their papers have open data after the year 2013. And we found that of about 4000 or so papers that we assessed manually about 20% of them have associated open data. And then we assess the quality of these open data sets for sample of these papers so 362 of them that had been shared by 100 profs or PIs. So important to note here that there are multiple papers per individual PI. And lastly, we found that 45% of the data sets were incomplete, and just around 50% of them were not reusable. So the data are slightly better than the ones I showed you earlier, but still about in the same ballpark. And so to answer our initial question, what explains the variation, we ran a multivariate mixed effects model to look at what factors affect the completeness and the reusability of these open data sets. We selected five independent variables a priori. All of this study was pre registered on the OSF. And so we looked at the year of an author's first publication as an indication of their seniority. We looked at the year that the paper was published, the gender of the PI, the open data policy of the journal where the study was published, and whether the corresponding author was sorry, whether the PI was the corresponding on the paper, the corresponding author on the paper, in addition to also being the first or the senior author. And again, what we found here is a little bit underwhelming. Nothing really seems to be particularly important in predicting the quality of open data sets, including a journal's policy. So there's some indication here, those black dots that more recent studies have slightly more reusable data than older ones, and that more junior researchers share more reusable data than senior researchers but there's really nothing to go to town on. So this would have all been quite disappointed but we did actually find something else that's quite neat. I'll show you this while wrapping up. So these graphs here show the data completeness and reusability scores for each one of the hundred PIs included in our study. So the PIs are ordered based on their mean completeness and reusability scores from lowest to highest. And that pattern becomes important when I'll explain this graph. So here, I've blown up the graph for data completeness so you can see it a bit better, and also because the patterns for data completeness and data reusability are essentially the same. And so like I said, because we have repeated measurements for each one of our PIs, we are able to decompose the variance in the response variables to look at among and within individual differences. What we find here is that there's quite noticeable inter-individual differences. In fact, PI identity explained the largest proportion of the variance in data completeness and data reusability. So what this means visually is that some authors consistently share incomplete data, regardless of the year when they published their career stage and the journal's data policy. And that might just be an indication that these people never want to share, right. When you look at the other end of the graph, we have PIs that do very much the opposite and consistently share high quality data regardless of whether they are asked to do so. And then we have everyone else that sits right in the middle here in this red box. So these people are all over the place. Sometimes they share well, and sometimes they don't. And I think this cloud of data points here is really encouraging because it means that these PIs are clearly capable of sharing well. They're just very inconsistent. This is my last slide. So maybe that's just because they struggle when data sets are complex, or it could also be that not all their students are properly trained in data management. And regardless of the reason this is all really positive because to me what these data suggests is that for these people who are actually the majority of the people, proper data sharing is not really a matter of will, it's a matter of competency. And so my takeaway from these data is that if we provide these PIs in the red box with the right incentives and the right training in data management, then I'm ready to bet anything that will be able to move them from the bottom of this graph to the very top. Thanks very much. Thank you so much, Dominic. So moving into the next talk will be by Kevin Joyel Demeray, and I will be sharing my screen as he's provided a prerecorded talk, but as you can see he is available to answer any questions that you might have. A project that evaluates how well covariates can reduce sampling biases in the context of convenience based samples. Sampling bias occurs when a study is conducted in such a way that different individuals within a population have unequal chances of participating. This can occur for many reasons. One way is that certain sampling strategies disproportionately reach certain populations and may also be more versus less successful and encouraging participation. For example, a well-paid large-scale web panel may lead to high participation rates, whereas a convenience-based sample of unpaid volunteers may suffer from lower rates. Participant characteristics can also factor in. For instance, we found that women are more likely to volunteer for research than men, and people may be more likely to participate in research if they have positive attitudes towards the topic on this study. For example, research on vaccination may attract more individuals with positive vaccine attitudes. Now the problem is that because we only have responses from those that actually participate, this will commonly induce various associations between the factors that influence participation itself through well-documented processes such as collider and selection-based biases. Now in this scenario, a convenient sample will overestimate vaccine attitudes, and we may observe that women show more negative attitudes than men towards vaccines even if no such association exists at the population level. So one strategy to eliminate or reduce bias is to adjust analyses for covariates in an attempt to disrupt various associations. Ideally, theory can gag which covariate to select, but most theories are not specific enough for this task yet. Consequently, researchers will often default to heuristics or norms such as controlling for demographic variables like age, education, or ethnicity, and different researchers will often choose different covariates to adjust for. So given this reality, the goal of the current project was to answer the following question. What is the general success rate of adjusting analyses for demographic covariates for attenuating sampling bias? To answer this question, we examined the degree to which covariates can help align results between convenience and more representative samples. We used data from the ICARE study, a pandemic-related survey that has been ongoing since early last year. In Canada, ICARE has recruited both convenient samples and more representative samples using web panels constructed using probability-based sampling. We used data collected through both methods at three time points. This allowed us to compare the degree to which the two sampling methods led to discrepant estimates across 33 different outcome variables. We operationalized sampling bias as the difference in means on each outcome variable when comparing the convenient samples to the web panels. For example, relative to the web panels, the convenient samples consistently overestimated people's vaccine intentions. In unadjusted analyses, we found that 73% of outcomes differed between the two sample types. We then examined how nine commonly used demographic covariates, such as age, gender, income, and race, altered estimated discrepancies between the two samples. With nine covariates, we could specify 512 combinations from adjusting for one covariate at a time to adjusting for all nine and everything in between. In total, we computed close to 17,000 models to evaluate the impact of these covariate combinations. We operationalized success as covariate combinations that decreased discrepancies between the two sample types compared to unadjusted models. What did we find? Overall, covariates decreased discrepancies 55% of the time and increased discrepancies 45% of the time. Additionally, we did not find any evidence that specific combinations of covariates performed well consistently. Most covariates showed fairly high chances of both increasing or decreasing discrepancies depending on outcomes. For 27% of outcomes evaluated, the vast majority of covariate combinations were successful at reducing discrepancies. Here we see a caterpillar plot for the degree to which participants in the two types of samples differed in the extent to which they avoided social gatherings. Each dot represents an estimate for a specific model that controlled each of the 512 combinations of covariates. 95% confidence intervals are shown around them. The solid black line indicates a sampling discrepancy of zero, which we operationalize as unbiased. In this plot, models on the left show smaller discrepancies between the samples and models on the right show larger discrepancies. The unadjusted model is indicated by the dashed line. In this case, 93% of models adjusting for covariates reduced the estimated discrepancy between the samples. So the inclusion of covariates was typically successful regardless of the covariate that was in the model. For 39% of outcomes, however, adjusted models were roughly just as likely to decrease or increase discrepancies. In this case, here is a plot of discrepancies for people's vaccine intentions. 50% of adjusted models led to decreased discrepancies and 50% led to increased discrepancies. For many of these models, most combinations of covariates did little to change the original significance level of the unadjusted estimate. In this case, the convenient sample overestimated vaccine intentions regardless of the combination of covariates being adjusted for. Lastly, for 33% of outcomes, only a small minority of adjusted models reduced discrepancies. The majority of adjustments instead increased discrepancies. For instance, an unadjusted model, the convenient sample, rated COVID-19 prevention measures as more important than participants in the web panel sample. Although this initial estimate was not significant. And 87% of adjusted models, everything above this dashed line, the discrepancy was increased. And often it became significant. So to conclude, we found little evidence that adjusting analyses for demographic variables in an unstructured, that is non-theory driven way would reliably reduce sampling biases. With reductions in bias occurring only 55% of the time across nearly 17,000 models. This is barely above chance level. I hope you enjoyed this presentation. And thank you for listening. Thank you for that, Kevin. And so, like I mentioned, Kevin is here if you all have any questions, please feel free to submit them during the end to the Q&A feature. Moving on to the next talk is another pre-recorded one from Joshua Wallach. And so I will get that started. Hello everyone, my name is Joshua Wallach and I'm faculty within the Yale School of Public Health. And today my lightning talk will be about the dissemination of clinical and health science research as preprints. In particular, I'm going to be focusing on the question of, should we be concerned about pre-printing clinical research. Now before moving on, I want to quickly, you know, say thank you to the funders and organizers of this conference. It's a great conference and community. And I'm honored to join the discussion today. So this may not be necessary, but just as a background, your preprints are preliminary reports of studies that have not been peer reviewed or published in peer reviewed journals. And there's a long history or a reasonably long history of pre-printing in other fields with the pre-print servers archive and bio archive allowing for studies in the fields of physics and biology and life science. And the uptake and use of pre-prints in clinical research has been a little bit slower. In particular, there have been concerns about the dissemination of clinical findings prior to peer review and the impact that non-peer review findings could have on the decisions that individuals make. However, in 2019, Met Archive was launched and this is the pre-print server for health sciences. You know, since that time, a number of these concerns about pre-printing and clinical research have been discussed and have been brought up. And one of these is that high impact clinical journals will view pre-prints as prior publications so that in clinical research, and especially in high impact journals, pre-printing will prevent publishing in the journals. And, you know, here I have an example of a journal policy from a journal with an impact factor of greater than 10 that says manuscripts that have not been posted on a pre-print server will not be considered by the journal. So again, there's these, these are some of the concerns that clinical researchers have. So in order to try to understand the validity of these concerns, we conducted a study looking at the pre-print policies in the highest ranked clinical journals so focusing on the top 100 highest impact factor clinical journals. What we found in this study published in JAMA Network Open was that the vast majority of high impact journals actually allow pre-prints. And, you know, the, you know, there's 13 or so percent that have a case by case determination process which don't outright prevent or forbid pre-printing of manuscripts. There was only one journal that prohibited pre-printing and that's actually the example that I showed on the previous slide. Another concern that has been raised is that clinical pre-prints are harmful and more likely to be withdrawn from the literature. So again that in clinical research, the dissemination of, you know, questionable findings are harmful and that, you know, there's going to be a large proportion of pre-printed studies that are eventually withdrawn from pre-print servers. And a number of high profile editorials have raised questions about why this is really an important issue for clinical research, findings from trials of interventions or observational studies that could have consequences to patient populations. And, you know, these editorials have called for additional research or empirical research to try to understand the both the benefits and harms of using pre-prints and clinical research. The team behind Metarchive looked at all of the submissions of pre-prints in the first year of Metarchive. And what they found was that only 0.002% of studies have been withdrawn. So 0.002% of studies had been removed from the Metarchive platform. And there's just some reassurance that there's not a large number of studies that are being withdrawn from Metarchive. And one of the, one of the third main concerns that are often discussed when it comes to pre-printing and clinical research is that the conclusions and interpretations of pre-prints are likely to change once they're published. The manuscript is posted on Metarchive. Once it's published in the journal, peer review and editorial changes are likely to have impacted what actually is reported and the findings are likely to be different. And, you know, these concerns have been raised again in a high profile editorials. Here's another one in JAMA talking about how in the JAMA journals, the conclusions and interpretations of research articles are likely to change substantially after they are submitted. And that this is a result of peer review, editorial assessment, post acceptance editing, etc. So to try to understand this, this question of whether or not pre-prints are likely to change, we conducted a study looking at the concordance between pre-prints and peer reviewed versions of the same manuscripts that eventually are published in the highest impact factor journals to see whether or not the conclusions and findings are likely to change. And among our sample, we found that pre-prints and peer reviewed versions of manuscripts were highly concordant in terms of primary endpoint. So we weren't observing a large number of primary endpoints that were being changed between the pre-printed version and the peer reviewed version. You know, although we found that only 53% of the results were concordant between primary endpoints. So the findings themselves. So worth noting that 21% of the pre-print publication pairs could not actually be compared because they reported non inferential findings or non numerical findings. So, although it's a it's a lower proportion of studies with concordant primary findings. You know this is a little bit more skewed because of the inability to actually compare pre-print and publication findings due to non numerical results. We're looking to see that when we were looking at the concordance and study interpretation so what the authors actually conclude based upon their results. The vast majority of pre-print and publication pairs had concordant study interpretations. 96 had very similar or identical study conclusions. I want to note that Med Archive is focused on transparently reporting some of the limitations related to pre-printing clinical research. In particular, the homepage of Med Archive said, says that there's, you know, caution needed when it comes to evaluating research, especially health related behaviors and clinical practice that have not been peer reviewed. The manuscript pages themselves say that articles are pre printed and haven't been peer reviewed, and they should not be used to guide clinical practice. So far the empirical studies related to evaluating concerns related to pre prints have suggested that one, many high impact factor journals actually do not prohibit pre printing prior to publication. So the number of manuscripts that are withdrawn from Med Archive is very low, and three, there's high concordance between pre prints and peer reviewed versions of manuscripts that are published in high impact journals. And although additional research is necessary to determine how concerned we should be about pre printing and clinical research, in particular to think about how the media talks about pre printed study results, and how concordance study findings are in the long run. The findings provide some initial reassurance about the use of pre prints in clinical research. Thanks for the opportunity to talk about this and I'm looking forward to discussions on Slack and outside of the conference about the use of pre prints in biomedicine. Alright, thank you to Josh for sending that in. I will now go ahead and move us on to our next speaker, which is a Pranajan Pathmandra, who will be discussing wrongly identified reagents in original research papers published in high impact factor cancer journals. Pranajan, the floor is yours. Just getting this slide so sorry. Can everyone see the slides. Yep, looks great. So my name is Pranajan Pathmandra and I am a fourth year undergraduate student at the University of Sydney. I'm here today to present some preliminary findings on my honors project on the presence of wrongly identified reagents in original research papers published across a particularly high impact factor cancer research journal. So in 2015, Friedman and colleagues estimated up to 50% of the funding in pre clinical research to be associated with data that was irreproducible. One of the main reasons being the presence of faulty or misreported biological reagents in reference materials that did not explain the described results. And so science which fact checks these biological reagents is very important in maintaining good standards of biomedical research integrity. One of the main class of biological reagents is the nucleotide sequence reagents which as you can see our strings of DNA or RNA sequence that are used across many different type of experiments. These sequences are like barcodes and their identity cannot be just cannot be read by just looking at them. So we in so in ideal publications, we have text descriptors that accompany these nucleotide sequences, telling us what their function is. However, given this difficulty of configuring sequence identity errors in these reagents can easily go unnoticed. The current fact checking process behind looking at reagent ISE nucleotide sequence reagent identities is to really take the sequence and input it into a search algorithm such as blast and or blast, which maps the sequence against a well annotated human sequence database and configures a verified identity on the basis of similarity. Ideally this verified identity should match the originally text claimed identity, but a mismatch means we have a wrongly identified sequence reagents. The commonly identified sequence reagents are associated with unreliable and often implausible results as the described methods containing these problematic reagents does not explain the observed results, making the results themselves questionable and potentially irreproducible. These reagents could also be reduced across future studies and thus these incorrect reagents could be propagated to downstream research, ultimately leading to a massive waste of resources and time. And ultimately this could also reduce the public trust in the scientific method and community, hence highlighting the need for practices of fact checking these biological reagents as essential to maintaining biomedical research integrity. Given this, Bernan colleagues looked at two cohorts of problematic papers and as you can see here in clearly in group one, the most of the papers were published across journals of lower impact factor. But in group two they identified at least one publication that contain these wrongly identified sequence reagents, and thus potentially concerning results to be published in a very high impact factor journal. And so really taking this forward, that was the aim of my current project, which was really our papers with wrongly identified nucleotide sequence reagents also published in high impact factor journals. To answer this we chose to look at the journal molecular cancer which is a part of spring and nature, and it displayed and it was it had an impact factor above 27 in 2020. We chose to analyze all original papers published across 2014 1618 and 20, meaning that we had to look at 500 papers that were published in that in that journal. Across the four years we analyzed of the 500 articles 68% that is 341 articles included nucleotide sequence reagents and thus was subject to fact checking. These 341 papers presented over 6500 individual sequence reagents of which fact checking revealed only 7% so that is 460 sequences were wrongly identified. However, these 460 problematic sequence reagents, or wrongly identified sequence reagents were distributed across a large proportion of the papers we analyze that is 41% of the 341 papers that we analyzed in molecular cancer across these four years, described wrongly identified sequences, and thus potentially questionable results when mapping these error rates back to as a proportion of all original papers published per year. We see that 28% of the 500 original papers published across these four years in molecular cancer, describe wrongly identified sequences, and thus could hold potentially questionable results, particularly what is most striking is 2020 in a year in which the impact factor was really high above 27, yet we see that 44% of all the original articles published in that year in molecular cancer contained on wrongly identified reagents and thus questionable results. The presence of such incorrect reagents is thus a concurring problem that affects research as recent as last year. In summary, we find that original research papers with wrongly identified sequence reagents are clearly not restricted to just low impact factor journals. As we as we found here in our preliminary results 28% of all original molecular cancer papers included wrongly identified sequences and thus question and poses very much a lot of questions to the observed results. This is a very highly unexpected and concerning result. And to prove and to examine this phenomenon further we will continue to analyze other high impact factor journals, but nevertheless what is clear is that the presence of problematic reagents is an alarming threat to biomedical research integrity. I'd like to end the talk by thanking my supervisors and their respective funding agencies. Thank you for listening.