 Randomist experiment is the golden standard of empirical research. However, there are ways that randomized experiments can go wrong. So just the fact that you randomized your study sample in the treatment and control groups and put those two groups into different procedures and then measure a difference in the outcome variable of interest does not necessarily imply that you have a valid causal claim. In this video I will explain a couple of problems that experiments may face. When we talk about experiments we need to remember that there are two important properties of an experiment. So two things that make an experiment. First we have the randomization here and then we have the treatment group and the control group. And this provides valid causal evidence if the randomization works and our sample size is large enough and there are no problems with the procedures. When we talk about validity of conclusions from experiments we need to consider internal validity and external validity. External validity is basically whether the results from our sample or our population generalizes to other populations of interest. And a typical problem in external validity is the use of student samples. So for example if we want to study how directors make decisions in companies and we study that through students who do a business simulation in a classroom the external validity is pretty questionable. Student samples are not always bad but we have to consider the context. For example if you want to study personal use of IT then at least I wouldn't have any problems using students because our students and general population are more similar in that than how students in classroom works is similar to our boardrooms for example. But there are also issues related to internal validity. Issues that make your causal claim questionable even for the population of interest. So for example we can study students in a class and only try to generalize to what those students would do outside the class and be not able to generalize because of these issues. There is a nice review of these issues in experimental design by serial onati and his co-authors in general operations management and they have this summary table that lists certain issues that they explain in detail in the article and they divide these issues into statistical issues and internal validity threats. I'll focus on the first set of issues because these statistical issues perhaps except for excluding non-compliers is more general. So if you don't take into consideration non-independence of observations then any inference with any research design is possibly invalid. So let's take a look at these issues here, these internal validity issues and I'll go through an example that demonstrates this on the next slide. So the first on the list is unfair comparison and demand effects. So that's basically two different problems. The unfair comparison can be understood with the poison and medication examples. So if you have your treatment group receives the medication and your control group receives the poison, the fact that the outcomes are different does not mean that the medication worked. It could also mean that the medication didn't work but poison just made people feel a lot worse or in an extreme scenario it might mean that the medication actually is harmful but poison is more harmful for people. So the important thing here is that your control group should be really neutral and not like this good and bad comparison. Another is treatment effect and demand effect. Demand effect relates to the subjects in the experiment trying to infer what the experimenter is trying to study and how the experimenter would like them to respond. And this is something that has been studied and there is evidence that this phenomenon actually exists even if people are not consciously trying to satisfy the demands of the experimenter. So that's the first group of issues. The second group is non-consequential decision environments and this is particularly relevant for experimental Vignette studies where we send surveys, two versions of surveys that describe the same scenario with small variations and then ask people questions about that scenario. So if you just fill in a survey for there are no consequences for you from your actions and it's not clear if you would respond the same way if their actions were consequences. I'll show you an example on the next slide. Then there is deception and deception does not necessarily invalidate the study but there are two arguments against deception. One is the ethical argument that researchers should not lie to their subjects. So if you deceive intentionally mislead your subjects then you are being unethical. There is some debate on whether being unethical this way is acceptable in some scenarios where the results will be very important to get. So there are some important studies in the history that have been done using deception and some of those studies like the Milgram's experiment would be considered really unethical now. Then there is another issue about deception. So if you have a lab where you invite people, particularly if you invite students there and you know that the students will be subjects in a couple of experiments during their studies if you deceive them and they find out that they were lied to in the first experiment how are they going to take you seriously in your second experiment. So the arguments against deception is the ethical argument and it's also the argument that we are kind of like spoiling our subject pool by lying to them. Then the fourth on the list is manipulation checks before the dependent variable. And the idea here is that the manipulation check, what it means is that if we for example give people medication and that is a kind of medication that people take at home. And then they come back for measurement a week later. We ask them did you actually take the medication? Because some of our subjects might have forgotten to take the medication and that needs to be taken into account in the statistical analysis. In practice that will be a case for using instrumental variables. Problems arise however if we do a manipulation check before the measurement of the dependent variable. And it is then possible that the respondents particularly if we measure do survey based measurement or some other kind of measurement where we measure people's attitude then the subjects may infer based on our manipulation check what we are actually studying and then trying to adjust their response accordingly. So let's take a look at an example and how these effects might manifest in a study. So this is a completely made up study. And this is an expert in a big net study. The idea is that we present two scenarios. One individual receives one of these scenarios in a survey but not the other. And this is randomized. So half of our informants receive scenario one, half of our informants receive scenario two here. And then we ask based on these two scenarios two things is the company performing ethically. That is our manipulation check and would you buy the shoes? So we have shoes that are less expensive than major brand shoes and you really want to have the shoes. You hear that this company uses child labor and you hear that this company is behaving very ethically. They have a corporate social responsibility program that they just announced. So how are these issues listed in the Lonati article manifested in this example? So first of all we have an unfair comparison. So we are not comparing a bad company against a neutral company. But instead we are comparing very unethical company against a very ethical company. So we cannot say that doing unethical things would be bad because the baseline is not doing unethical things. But the baseline is doing good for the society. Also we cannot say that CSR programs will be helpful because the baseline is not no CSR. But it's very unethical behavior. So that's an unfair comparison. It's a poison and medication comparison. If there's a difference we don't know which one causes it. Then there's a demand effect. So if you read this short vignette you see that this is just basically facts. And then there is this statement that stands out even if it wasn't bolded. That this company is using child labor. So that is not something that you would perhaps know if you were to buy athletic shoes. And then there's the other thing here that this company is using CSR, implementing a CSR program. That is also information that you probably wouldn't know or wouldn't notice even if it was given to you in a broader context. But in isolation this stands out and it is clear that the experiment here is about ethics or corporate social responsibility or something like that. And that guides our responses. So if we see this kind of vignette here then we pretty much know that the researcher wants us to answer no here. We would not buy these shoes and same here the CSR would imply to us that the researcher is studying social responsibility. We are supposed to say that we buy these shoes even if they are less expensive for some reason. So that's the demand effect. This is also a non-consequential decision. Why it's non-consequential is that this is just imaginary money. So let's say that the brand name shoes cost 100 euros and these cheaper shoes cost 70 euros. If you really are short on cash and you need new shoes you might think that well this time maybe the company will be better in the future. It's just this time that I buy these shoes from this slightly unethical company. If there's real money on the line people may behave differently than when it's just a question of what would you do in this imaginary scenario. Then the final thing in this example is the manipulation check. And this clearly demonstrates that the manipulation check question is here. Is the company either in scenario one or scenario two behaving ethically. That really gives out the purpose of the experiment. If we read this manipulation check and which purpose of this check is to basically ensure that we have received the manipulation. So we have noticed that one of these is more ethical than the other one. And this underlines that this is a study about ethics and then people will respond accordingly saying yes to the ethical case. No to the unethical case because that is what they think that the experimenter wants to see.