 Randomized experiment is the gold standard in empirical research. However, experiments are not always feasible due to, for example, cost or ethics reason. And for that reason it is also important to understand that there are other designs that kind of like mimic experiments that we could apply when a true randomized experiment is non-feasible. In these techniques or these designs fall into pre-experimental designs and quasi-experimental designs, which are sometimes referred to as natural experiments as well. Let's take a look at what these designs are. So the idea of true experiment is that we have these five characteristics. First we have a random assignment, so we randomize people in at least two groups. So we have the treatment group and then we have the control group. Then we manipulate the main independent variable, whether one group gets treatment, other one doesn't or some other manipulation. We measure the outcome from both groups and we compare the difference. And fifth importantly, there must be experimental control in the sense that these groups, they undergo the same procedures except for the treatment. So for example in medical trials we have the medication pill and then we have a placebo pill for the control group so that both groups actually go the exact same procedure, but only one of the groups get the actual medication in the bodies. And importantly experiments do not need to be conducted in labs. So experiments can be done in the field, so you can go outside the laboratory, go to the lobby of the university, go to a real company, go to the city and do your experiment there. We can also do field trials and trial refers to trying out something. For example if we have schools we could get 100 schools to participate in a trial of an anti-bullying program and then half of those schools would implement the program, other half would be in the control and then we would compare the outcomes after the program is done or some years later. So that's the field trial. Also we can do experiments in surveys. Survey items are often tested by doing these two different forms, slightly different survey forms and then we have our sample, we divide it into two randomly and then we check for small differences in how we frame our questions cause differences in actual response. This kind of testing is also done quite often on internet, so internet companies do this, what they call A-B testing, so people receive two different user interfaces randomly and then the company checks if these two groups of people behave differently with the software. An important class of these randomized surveys is the experimental Vignette study and these studies present the informants scenario and some characteristics of the scenario are varied and then the two variations are randomly assigned to the informants and then we check if there's a difference. Again the important part is not that we have lab but the important part is that we have treatment and control random assignment and then experimental control so that the treatment and control groups only differ in the manipulation that they receive. Let's take a look at what other designs we have but before that or other designs. So experiment is the gold standard but we have designs that kind of mimic experiments. These are sometimes called pre-experimental designs and quasi-experimental designs. What exactly differentiates between quasi-experimental and pre-experimental design depends on which definition or which book you look at because the difference is not so clear cut. According to this tires definition pre-experimental design is a study where there is a treatment and observation of an outcome but there is no control group so we can only compare either the group that receives a treatment against itself from previous time point or don't do any comparisons. And then quasi-experimental design adds to that a control group but it lacks randomization and then the true experimental design of course has the five features that I discussed on the previous slide. Let's take a look at different designs that we can construct and this is the gold standard, the experiment, we have the treatment and control, we have randomization and this experimental design can be expressed with this shorthand. So we have R randomization, we have X the treatment, then we have control, the group that doesn't receive the treatment and then both groups are measured after the treatment. So this is a way of writing down an experimental design or quasi-experimental design in a short way and I'll be using this notation on the next couple of slides. Let's take a look at what these pre-experimental designs are. And Singleton Estrates lists three experimental designs, we have the one short case study, we do something X, we observe what happens O, for example, if I implement a new teaching strategy on my course and then I measure the student satisfaction of the course afterwards that will be one short case study and it's problematic because let's say that I receive a high ratings for my new teaching approach, that doesn't really tell whether the approach works or not. It's also possible that the student group just happened to like the subject of my course and the outcome had nothing to do with the teaching strategy that I applied. A second slightly better strategy is one group pre-test-post-test, so let's say that we are doing a medical trial and we measure the health of the individuals before the treatment, then there is the treatment and we measure the health after the treatment and we compare the difference. But this is not a valid causal claim or valid causal effect because it is possible that there would have been a difference in any case. So it's when people are sick, they get better spontaneously without any medication and also some people that are not sick get sick during the treatment. So we need to have the comparison. A third is static group comparison. So the idea here is that we have a group that receives treatment, we have a group that doesn't, but it is not randomized. And even if we have a difference here after the treatment, the problem is that we don't know if that difference is due to the treatment or if it was a difference that existed even before we did anything for the two groups. Let's compare these against experimental designs. So these are the three experimental designs or three main variants of experimental design that singletonous traits present and we can easily see that the one-shot case study can be converted to a true experiment by adding randomization and a control group. So this is the post-test only design and this one group pre-test, post-test can be converted to a true experiment by adding randomization to treatment and control as well. So typically when we go from pre-experimental to experimental, we add randomization and we add a control group. Then there's the third design, the within-subject design. These are rare in social sciences because the idea here is that we have, for example, two treatments that we compare and every individual receives both treatments. And this assumes that the treatments we can apply or administer the treatments so that one treatment does not interfere with the other one. And they are rare so I don't address them anymore on this video. But that's basically the typical experimental designs and any experimental design can be expressed as a variant or extension of these. Now what are quasi experiments? So quasi experiments are somewhere between the pre-experimental and actual true experiment designs. So we may have a non-random assignment or we lack some of the O's and X's in those previous designs. Some of the designs that are typically presented in text or quasi experiments is the separate sample pre-test, post-test design. The idea here is that we have randomization but we can, everyone receives the treatment at the same time but we can measure this and the constraint that everyone receives the treatment. So let's say that there's an important medication that people just must receive. So it would be unethical to not give it to them. And we can still test the effectiveness of the medication by randomly measuring half of the people before the medication and half of the people after the medication. Let's assume that we are here constrained that we can only measure each person once. So typically we apply quasi experimental designs when we are not allowed to or cannot afford to run a full experiment. Of course it will be better if we measure the outcome for both groups before and after and even better if we leave this as a control but we could not in this case. Then it's possible that we have a non-equivalent control group design. So the idea here is that we have pre and post measurement. We have treatment group X, we have control group non-X but we don't have randomization. So perhaps there is self-selection. Perhaps somebody else selects our treatment group or our control group. And whether we can make a causal claim here depends on what kind of assumptions we are willing to make. And then we have interrupted time series design. And this is sometimes classified as a pre-experimental design because it doesn't have a control group. But here the idea is that we have a time series of repeated measures. Then we have something happens or something or we administer a treatment here and then we follow the time series again. The idea why this could be considered as a quasi experimental design is that because of this time series nature here we can use the individual before the treatment as a counterfactual group for the individual after the treatment or the group after the treatment rather. So we can estimate the trend here before we can estimate the trend after and then we can based on those two trends we can infer what would have happened if X wouldn't be here. So that's an important quasi experimental design. These quasi experiments are sometimes referred to as natural experiments. The idea of a natural experiment or what makes a quasi experiment is a natural experiment is that there is, for example, a natural treatment and let's say that there is a company that happens to implement a paper performance program but it only does it for half the people for some reason and they are assigning randomly. So someone is randomizing but it's not us. Or it could be that there is a new policy implemented here, X and that policy is implemented independently of what happened to the outcome variable. So the idea of natural experiments is that there is something that happens to either a time series or one group but not the other and the important part in natural experiments is that the X here, whatever happens must be exogenous or at least it must be that we can make reasonable assumption that it is exogenous. So how do we then analyze, how do we find or how do we design quasi experimental studies? Typically when you do your study, you start with a research question, then you do a research design, you design the experiment, you design the treatment condition, you design the control condition and then you run the experiment. With natural experiments, this is not about design as much as it is about discovery. So if you want to do a natural experiment, then the typical way is to go and try to discover sources of natural variation in the X that you're interested in that hopefully is exogenous or at least sufficiently so that we can use the natural or quasi experimental techniques and the data analysis techniques that are used with these designs. This article by Sivak and Santoni recommends that a good way for discovering natural experiments is to look at natural experiments that have been published before. So perhaps there are some scenarios or some phenomenon that happens to generate these kind of natural experiments, then you can go and find an instance of that phenomenon and then study yourself. But these are difficult to find and perhaps that is one reason why they are not so common in management research. When you have discovered a potential natural experiment, you need to decide how you go about analysing the data. And this paper also provides a nice flow chart. And the first question that you need to ask when you have a potential natural experiment is is the assignment to treatment random or as if random? So as if random means that we can assume that it is so close to being random that it doesn't make a difference if it's not exactly random. For example, if people get to self-select into receiving medication or not receiving medication and then we measure the outcome, is this assignment random or not? To answer that question, we need to consider three different things. First, do the people have information? So do they know that they have the option to go into these two treatments? Are they aware of their health, which is our dependent variable? And if so, if they have the information, they know their health, they know that they have two options. Then the second question is that do they have incentives to use this information to have a preference to go into treatment group or control group? So for example, in the case of medication, we could assume that if people know their health, they know that the medication probably works and they know that they have the option of not having the medication or having the medication, then perhaps those people who are more sick have more incentives to be in the treatment than those people who are less sick. And then the final thing, do they have capacity? So can people actually make a decision? So perhaps you are not taking your medication yourself, but someone else is giving to you in which case you wouldn't have the capacity. If individuals have information, incentives, and capacity, then we can infer that the treatment probably is not as if random. If one of these is missing, then the as if random as Samsung is probably more defensive. The second thing that we need to consider is do you need to change treatment status? So what that basically is about is compliance and non-compliance. So if an individual chooses to go to the treatment condition and receive medication, do they actually take the medication? Or the other way, if an individual chooses to go to the control condition, is it possible that they somehow get access to the medication? So that's probably not as relevant as the first kind of non-compliance. If we have as if random and if there is no non-compliance, then we apply just normal analysis that we would apply for experimental design. So just normal regression analysis or t-test between two outcomes, depending on whether you want to add control variables or not. If there is non-compliance, then like in any other experiments, you must use instrument of variable design. So we use the treatment as an instrument. This is our explanatory variable, and then we have the outcome variable. And we use instrument of variable estimation techniques. If the treatment here is not as if random, then we have another question that we need to ask. Is it possible that there is a variable that determines, either deterministically or probabilistically, whether an individual goes into the treatment or not? And if yes, then we can apply regression discontinuity design. If no, then we basically have a correlational design. But this is a very useful flow chart, because it really simplifies the choices that you have to make, and the article itself explains the reasoning and what kind of things you need to consider when you make these three different choices here. And of course, if we don't have this possibility for regression discontinuity design, if we have an exogenous source of variation, like in observational studies, we can always use an instrument of variable design. So these quasi-experimentale designs are very usable, and they may have higher external validity than experiment, while still having quite strong internal validity. Unfortunately, they are not very common in management research. One thing that may explain this, that natural experiments are difficult to find, but this is certainly something that any researcher who works in the field should consider.