 Comod method variances is commonly a concern in cross-sectional studies There are statistical techniques that are supposed to address this issue. Whether those techniques work or not, that's another question. In this video I will give an overview of some of the more commonly used technique and I have another set of videos that talk about the specific aspects of different techniques and they're specific problems. Here, I just provide an overview of what these techniques are based on and what kinds of principles we can generally apply to deal with common method variance problems. The idea of common method variance is that correlations between our measures are driven by the measurement instrument or measurement process, instead of being driven by correlations between the constructs. In our hypothetical example here, we have three measures of company innovativeness, three survey measures, three measures of company success, and we find that the scale score of innovativeness and scale score of success correlate at point three. Does that mean that the constructs innovativeness and success are correlated, or is it possible that these indicators actually simply measure how positively the person thinks about the company, how commonly the person responds to the right-hand side of a scale versus the left-hand side, or something else that relates to the measurement method instead of relating to the constructs of interest. This is the method variance. Typically, statistical models of method variance use this kind of framework, so we assume that all the indicators depend linearly on these constructs that we measure, and they also depend on a single source of method variance. This model, that is the basis of most of the techniques that we apply for dealing with the issue or doing diagnostics, has been criticized as being not very realistic. So this model basically assumes that there is this one source of variation that affects all items, and of course if you have things like priming effects such as how you answer to I4 affecting how you answer to I5, then this kind of model would not capture it. So this model has been criticized, and before you apply this kind of models in your research, you should understand the criticism. This article by Spectre and all other work that he has written about method variance is very useful in understanding the limitations and understanding how you can do more thoughtful modeling of measurement effects. But this is a commonly used technique, so it's fairly important to understand what are the drawbacks of this technique or the family of technique that apply this kind of single factor to be able to review work done using these techniques. Then how big of a problem is common method variance? There is actually disagreement on this. Podsakov's paper highly cited in 2012 puts common method variance states that it's a big problem by looking at these estimated percentages that are, for example, code and buckly says the variance of indicators 42% is attributable to the constructs 26% to the method, which would imply that the method variance is almost a bit more than half of the actual trait variance in the indicators. That would be a big problem. But then again these studies have also been questioned, and whether these studies have used realistic models to do the variance decompositions, we can always ask that question. The bottom line is that if a reviewer challenges your study based on a method variance problem and sides this evidence, then you need to either demonstrate that this evidence is actually not valid, which is difficult to do. Or you need to show that the issue is not relevant in your study or does not affect your study, which also is difficult to do. So the method variance problem is something like when someone knows that you may have it, then you have a set of bad options available. We don't know whether it's real or not, but there is evidence that can be cited that supports the claim that method variance is a big problem. Method variance issues are also a common reason for rejection. Of course, method variance problems, they correlate with other problems. So weak designs, typically if you want to have a causal claim, you need to have time difference, time delay between the cause and the effect, so that the cause is measured now and the effect is measured after an appropriate time period. If you do a cross-sectional study, then of course the delay does not occur, so it's difficult to establish the direction of causality. But these tend to be one of the reasons for rejection. This journal does not categorically reject any studies with Poisson method variance problem, but it's like one of the main reasons among other reasons that lead to rejection. How do people then address method variance problems? Of course, an idea scenario, you have a study where you design it in a way that method variance is not a problem. For example, you measure innovativeness using a survey, you measure performance using accounting data. It will be difficult to argue that those are affected by the measurement method because they are measured using different methods. But there are a couple of post hoc techniques. These are techniques that you can apply after data have been collected. And these can be divided into two categories. We have techniques that detect method variance problems and techniques that detect and control for method variance problems. So the first category of these techniques or first class of techniques is what I call as correlation techniques. And these correlation techniques basically you take the normal data correlation coverage matrix or whatever data that you have. And you only have measured those variables of interest that you have and you only have used one measurement method. So you're basically trying to explain correlations using different factor analysis techniques. And techniques that belong to this class are harmless single factor test, single factor converter factor analysis and various parts of correlation procedures and unmeasured later method factor design. I'll talk about these techniques in a moment. The second class is techniques where you have actually measured the source of method variance or you have marker variables. If you think that your measures are subjective to social desirable bias, one way to deal with that issue is to include items that measure that bias specifically. So there are different measures for how people like to respond to the right hand side or left hand side, social desirable, the self leniency, other kinds of things. So you can measure the causes of the bias if possible. Another strategy is to apply a marker variable. The idea of a marker variable is that you measure a construct that is theoretically unrelated to the constructs that you study. For example, one marker that people sometimes apply is whether the person likes the color blue or not. And it's very difficult to see how much you like blue would be correlated with, for example, company performance. And the idea is that you measure something unrelated using the same scale format and same measurement instrument. Then there are interesting constructs. And if those interesting construct and marker are correlated, then that correlation is solely due to the method. And that can be used to estimate the method bias. So this is a second class of technique. A third class of technique is multiple method techniques. And this is not very common. Multi-tread, multi-method matrices will be here. For example, you can measure something using a pen and paper survey, and then you can call back the informants and measure the same questions using a telephone interview. So that's one way, not very common. Then we have instrumental variables. Instrumental variables are a general solution, but they may not be very realistic. More generally, these techniques can be characterized into two more categories. Questionable techniques and impractical techniques. So then you need to basically, choosing a technique is to choose the least questionable practical technique. The instrumental variables are not practical. The reason why they're not practical is that when you do a survey study, typically all your data comes from that survey. And if you think that the survey as measurement instruments affects the items, then your instrument variable would be something that comes from a source other than the survey. Because if your instrument comes from the survey, then it's very difficult to argue that that instrument would not be affected by the biases that the person has when they answer a survey instrument. So typically the instruments would need to come from databases and you need large numbers, so not very practical to do. Then we have impractical and a bit questionable multiple method techniques. Impractical to call all the informants or impractical to establish multiple methods. And these techniques also make assumptions that are sometimes difficult to justify. Then we have correlation techniques. These are very questionable. I talk about those techniques in another video, but the bottom line is that these models are very seldom identified and they're not identified because you cannot check whether correlation between two items is because they measure correlated constructs or whether that correlation is due to the measurement procedure. So from a cross section with no markers or no measured sources of method variance, you simply cannot identify what is the source of correlation. Marker variables and measured method variance techniques are a bit less questionable whether the simplistic model of a single factor is appropriate. That can be called into question, but more thoughtful way of constructing your models where you model different sources of bias and you measure them with direct measures or markers. Maybe something that you can work with. However, particle with markers, there are some issues about model identification that literature has not really addressed this far. I'll talk about those issues in another video. Let's take a look at what these techniques do using a simplified example. So we have the simplified single factor model where we have two constructs and then we have a method here and this is actually a by factor model with correlated minor factors. The link between these method factor models and by factor models is not very strong in the literature in the sense that this method variance literature seems to develop largely detached from the by factor literature, but these are the same model. Then before we proceed, it's important to again repeat that this model itself has been called a bit questionable because there are sources of method variation that do not follow this kind of simple model. For example, item context effects, implicit theories and others, this model would do nothing to address those sources of bias. Post specters work addresses the problems of these general approach. So not all measures are affected equally and there are more than one source of possible bias and the evidence of existence of general bias is weak and questionable anyway. So that's a post specters argument. So he does not believe that this model adequately presents any measurement method effects. So that's why he calls it a myth. He does not say that there is no such thing as method variance. He just says that it's a myth, that it's a single source. It is more complicated than that. Let's take a look at these postdoc techniques for addressing method variance and let's start with the correlation techniques. The correlation technique is basically a technique where we take the original indicators and we simply fit a different model and we see what happens. This is the Harman's single factor test and this is basically the first test that people learn and it's not a useful test at all. So we simply take all the data that we have and we fit a single factor model. If the single factor model fits well, then if the single factor model explains large amount of variation in the data, then we conclude that method variance is a problem. If not, then we conclude that it's not a problem. This technique has two main problems. One problem is that it's not clear how much variance the method should explain for that to be considered problematic. So that's one problem. That's a more obvious problem. A less obvious problem or the more severe problem is that this method factor also explains the between-construct correlation. So if you consider this as like a by-factor model, then the general factor and the by-factor model basically explains what is the correlation between the minor factors. And that's what we have here. So if we have two highly correlated constructs but no method variance, this model would say that the method factor explains the data to a large degree because the items indeed are highly correlated because they measure two different things. So we cannot say from this model whether the variance explained by this factor is due to the method or whether it's due to the two highly correlated constructs. This is generally a technique that people recommend against. But nevertheless, because it's simple to apply, you see it applied quite a lot. There is one particular variant of this technique that is even more problematic. And sometimes you see this technique applied with a converter factor analysis. Then researchers check the model fit indices and they conclude that the model doesn't fit well. So what is the null hypothesis being tested there? It is that the method factor explains all variation, all covariation of the items in the data. So what we are basically testing with this converter factor analysis model of single factor is that there is no construct-related variance whatsoever in the indicators. And it is purely method variance. And that is of course a rather extreme case. So if you have like 40% construct variance, 20% method variance, that would be a problematic case already. So you don't need to have like 100% variation explained by the method for that to be considered a problem. So no trade variance in the indicators, that's what the converter factor analysis tests. So this is not a useful technique. And Podsakov's paper, for example, says that as well. They say that this is commonly used, but it's not useful. Unmeasured latent method factor, this is considered a lot more rigorous. And it's applied in converter factor analysis. But this too is a bit problematic. And how people usually apply this is that you constrain the first indicator of the method factor for scale setting, but you also constrain the other loadings to be the same. And that is done for identification purposes that are typically not explained in more detail. So what's the problem with this model? Well, there are a couple of problems. The first problem was that yes, there are identification problems. I'll talk more about these problems in another video. This is actually a great understatement. So this model has serious identification problems. And I generally would not trust the results of this kind of model ever, because you simply cannot identify whether the correlation between A indicators and B indicators are because of A and B correlate or because of the method factor. For the conditions under which this is identified or strongly identified are something that you wouldn't normally encounter in research practice. Another problem with this model, of course, is that the single factor model is unrealistic. But nevertheless, this is often considered the state of the art. And how people actually apply this technique is also problematic. Quite often people apply this and they conclude that the model does not fit better than the model. Without the single factor and then they conclude the method variance is not a problem. Without even interpreting how much variation the method factor explains. Second, when you constrain these indicators to be equal, you are saying that method affects all indicators equally. That's unrealistic assumption. And it also pretty much guarantees that this model will not improve the model fit, because you are including a factor that should not fit. So you are basically including a misspecified model. So you are artificially creating a scenario where this model cannot fit better than the model without the common method factor. And basically your test is just going to indicate no problem if you applied that. We all have another video of how you actually should apply and interpret these techniques. Then you have a marker variable techniques and measured method techniques. And the idea of these, we have first partial correlation procedures. And the idea of these partial correlation procedures is that we somehow estimate the degree of method variance. For example, we take a marker indicator and then we take an indicator of interest. We check their correlation and that correlation should be zero if there is no method variance in the data. Then we take that correlation, we subtract that correlation from the sample correlation matrix from every other correlation in the sample correlation matrix. And then we have the adjusted correlation matrix, we use that for our analysis. This is equivalent to having a single method factor that loads on all indicators with the variance fixed before the analysis instead of estimating from the data. So this is basically an unmeasured method factor design, except that we fix this variance based on an estimate that we get from a correlation instead of the data. So it's a bit problematic that we first take one data set and then we calculate the value for this variance and then we re-estimate the model using the same data set. It would be a lot better to estimate everything at one go. These methods come in different variants. So with the partial correlation procedures, we can have the partial correlation, we can calculate it based on a measured method. So we can, for example, measure social desirability and check how much our social desirability measures correlate with very interesting study measures. And then we have marker variable techniques, so I explained that already. And then we have a general factor technique, which basically estimates a single factor first and then you take that single factor out from the data, you parcel the effects of that single factor from the data and you estimate the model using the residuals. So this general factor technique is basically a combination of a single factor test, which is not a useful test at all and the partial correlation technique, which is also questionable. So it's a combination of two questionable techniques. Then we have measured latent method factor design. This is a less questionable technique. So the idea here is that if we have these measures of, for example, social desirability bias, then we can add a social desirability factor here and model how much social desirability affects the indicators. This is identified and it's actually a bifactor S minus I model, talked about that I'd and co-authors discussed. And it's, as a model, it works really well in terms of identification, but whether it's a realistic way of modeling the measurement effects or whether you need more factors, that's debatable. But general principle, if you have a single source of method, then this is the idea way of doing so. However, quite often these markers, these indicators, these M indicators are not measures of method variance, but they are markers and this is problematic for reasons that I'll explain in another video. So this is called measured latent method factor in Pojakov's paper and it's a pretty useful technique as long as you have just the one source of method variance. Then we have multiple method techniques. So you collect the same data using a mail survey and telephone, for example. These are not very practical because you need two data collection techniques. And in terms of how the model works, we have different measures. So let's say X1, X2 and X3 are each measured using a different technique. X4, X5 and X6 are each measured using a different technique and then we have these method factors that are techniques specific. This works, this particular model is probably not added for that, but as a general principle this works if you can get the multiple indicators. But there are issues that make this a bit questionable. And well, there is identification issues. So this is not always identified particularly if you allow these method factors to be uncorrelated, then this is not always identified. And then the second, the correlated uniqueness models here, this is basically the same model as this one before here, but it has method factors that are constrained to be uncorrelated with one another. So this model with correlated error terms is the same as a by factor model as I explained in the video about common factor model diagnostics. And whether the uncorrelatedness of these method factors is something that you can support theoretically, that's a bit questionable. So this assumption of uncorrelated methods is difficult to justify. So a multiple method generally are not practical. Then we have instrumental variables. So instrumental variables are a general technique for addressing measurement error or endogeneity or whatnot. The problem with instruments is that it can be difficult to get the instruments. So the idea of an instrumental model is that you actually don't measure the methods of variance source. So all other techniques that try to address method variance try to model the source of method variance. Here we don't make any assumptions about method variance. So we are not modeling method variance, so we are not doing any assumptions about that. So there are no questionable assumptions about method variance. Instead, we allow the construct A and construct B to have a correlated error term and then that is identified because of the instrument. The problem with instruments is that while this is a statistically sound approach is that the instruments must be measured with a different method than the interesting variables. Otherwise, how could you argue that the method cannot affect the instruments which you need to do to establish the instrument variable exclusion criteria? And quite often if you have let's say five different constructs that explain the dependent variable then you need at least five instruments. So you need large numbers of instruments and the instruments must come from another source and the instruments make this a bit impractical technique in practice. So the summary. The post-talk techniques are addressing common method variance. We have correlation techniques. These are questionable and ineffective. So basically if you did not consider method variance problem in the research design states there is really not much that you can do about it. I'll talk more about why these are problematic Then you have these things that you need to consider in the research design states. Using markers or using a measured method variable, method by a source is probably the least bad practical alternative. Having multiple methods or having instrument variables is difficult to accomplish in the research design but this is simple to do by adding just a few more questions that gauge the effect of the measurement method in your data into your survey form. However, there are arguments against this simplified method that you should take into account and one particular good source is specter and co-authors in 2019 and they talk about a different modeling approach. The idea of their modeling approach is that when you do these method factor models you should start modeling the method effects by first reading studies about the measures that you apply and evaluating the measures themselves to see what sources of method variation can affect those items. Then you add method factors for each source. Instead of having this general factor that affects all items on which most of these techniques are based on you do a more thoughtful modeling approach. Practical advice on method variance. First, the best way to avoid common method variance and common method bias is to avoid the problem in the first place. Procedural remedies and multiple sources are best. Consider the mechanism for method variance and its expected effect and build your model to model the mechanism to assess how large is the effect. Which effects are present in your study are the item context effects, priming effects, social design effects and others. Podsakoff and co-authors provide a long list of different methods, different sources of bias with references on what you can read to learn more about these bias sources. Then consider evidence of the strengths of these effects from the methodological literature. So what evidence is there to support that there actually are method variance sources that affect your indicators. And then make informed decisions and evaluate the impact based on this evidence. The correlation techniques, postdoc techniques simply do not work. They are indefensible. You could perhaps get your study published even with these techniques, but you should really say that these results are preliminary and you should include evidence that prior research that is done with more rigorous techniques has shown that your items that you apply have not been suspect to method variance problems or the method variance effect has been small in other contexts. Then maybe your results could be trusted. But as a general rule, the post hoc correlation techniques simply don't add much value. I'll talk more about those in another video. Correlational markers and multiple methods, marker variables, measured method variance sources and multiple methods can be effective, but to be able to apply this technique you need to consider those already in the research design states.