 In this part of the course, we will speak about the problems that can occur when science isn't done quite the way it should be. We will also speak about steps that are being taken to reform science, so such problems are ameliorated. First, let's take a look at how sometimes things don't go quite as they should in science. There's one important problem we have to consider, publication bias. Publishment bias refers to the tendency for experimental evidence to be published and made available to the public, only if it shows a result that is exciting or intriguing or fulfills certain statistical criteria. Publications, however, are what researchers' evaluations are based on, only if you publish a lot and in good places. Can you have hopes for securing a research position and keeping that job? Problematically, this means that other evidence, such as evidence that doesn't support a popular hypothesis, might be more difficult to publish. Therefore, researchers sometimes simply abandon such evidence because it is difficult and time-consuming to find a way to publish it. That means that the scientific literature becomes systematically distorted and consequently our basis for making inferences are shaky. When reading an article from a literature that is biased, we can't be sure how many refutations of the effect that this article shows have not been published. Therefore, we can't be sure we can trust the conclusions drawn from this article. This means publication bias is a problem for our confidence in the literature. This next phenomenon, harking, is related to the problem of publication bias. It also deals with trying to increase the number of interesting and easily publishable results you'll find in your experiment. To understand harking, consider this example. Let's say you have a fruit basket. You close your eyes, someone rotates the basket, so you don't remember which fruit is where, and then you reach into it and grab a random fruit. You take it out, look at it, and you have a banana. Now, if after randomly picking a banana, you say, well, I knew it was going to be a banana all along, and tell others, if you do this too, the fruit you will get is going to be a banana, then you are harking. It is obvious that this isn't right, isn't it? Harking means hypothesizing after the results are known. Sometimes, when we run an experiment and find a certain result that we have not actually anticipated, we may feel tempted to come up with an explanation for this result. And treat it as if we had expected this all along. Now, why is this a problem? We again distort the body of evidence available. Instead of admitting that we found evidence for something we didn't expect, we pretend like we had planned to test the theory that would lead to the effect we end up finding all along. Another related and potentially even more problematic practice is what we call p-hacking. P-hacking means that researchers conduct multiple statistical analysis about the same question and then only report the one that works and shows them the results they were looking for. For example, to p-hack your experiment, you could be flexible about the subjects you exclude from your statistical analysis. Maybe if you leave out five certain participants who didn't quite behave the way you expected, the analysis will then indicate that the effect that you set out to find seems to exist. However, this practice is problematic because instead of taking the data from the experiment for what it is, conclusions are based on artificially distorted data, data that is made to speak in favor of your hypothesis. The insights based on this data can then no longer be trusted. The last problem we will address here regards the number of observations in the experiment. We spoke before about the fact that it is important to sample data from a lot of participants. Let's look at what happens if you do not do that. If a test has low power, that means it does not have enough participants to detect an effect if that effect is really there. We might miss out on a positive test result simply because we do not have enough data points to find it. That alone would already be unfortunate. However, any test, as you remember, can make mistakes. This is also true for our experimental test of our hypothesis. There's always some probability that our experiment shows a certain effect even if it is not really there. Having low power makes it more difficult to differentiate between effects that are really there and those that only seem to exist. The proportion of false positive test results to true positive test results becomes worse. This means that experiments that are underpowered are not very informative because we cannot draw strong conclusions from a number of experiments about the same effect if they have low power. All in all, those are some pretty serious problems for scientific research. They can make you wonder to what degree you can rely on scientific information that is published. Thankfully, there's also a reform movement trying to counter these problems by changing the way science is done. So it'll become more open and transparent. In a way, scientists are doing an experiment on themselves to improve the work that they do and that in the end becomes available for you to judge and use. One of the key contributions of this reform movement is called pre-registration. This means that before running our experiment, we decide on the hypothesis we want to test and how to test them and register this information in an online database. After we've run our experiment and want to show it to others, we can therefore prove that what we tested was what we wanted to do all along. This means that what you see in an article that reports a pre-registered study, this is a signal for the authors having committed to their hypothesis before seeing the data, which helps us build trust in the experiment. Registered reports are another contribution by the reform movement. Registered reports are a publication format where a scientific journal agrees to publish your experiment before you've collected the data based on evaluating the methods and theoretical basis alone. That means that publication for these types of articles is not contingent on how exciting the results of the study are. This helps us ensure that the body of evidence that enters public knowledge is less distorted by the tendency to publish only exciting results. Moreover, the reform movement also encourages researchers not only to work on new experiments, but to also rerun or replicate previous experiments. Replications help us understand how good the evidence for certain finding is, whether it's robust and will predictably appear again and again. Replications are undertaken by individual researchers, but also by big groups of researchers who pool their resources to systematically rerun important studies. This helps us understand on which findings we can indeed rely and which ones we disregard because we find them unreliable. Finally, the reform movement advocates what is referred to as open science. This broad term means making science more open in many ways, from sharing materials and code to data and making publications available openly to the scientific and non-scientific audience alike. Transparenly conducted research makes it possible for anyone to follow up on it. If all materials used are available openly, you, me or anyone else can look at them freely, think about them, build on them and conduct their own research based on them. Therefore, open science helps us make sure that the insights we rely on are indeed reliable and can be developed further in a cooperative effort. In this part of the course, we have seen that there are some troubles that the academic system is facing and that could make us worried about whether we should trust scientific studies. However, many scientists care a lot about improving the system and therefore come up with new cool ways to change the way science is done. For the better.