 This will be more like a log kind of video and I'm shooting this from our basement for where you normally shoot my Teaching videos and this is there are the green screen, but I'm I'm not wearing my hoodie because I want this to be more of a Conversation article and the reason why I'm doing this video is that I've been asked two interesting questions recently And I thought that the answer to these questions might be useful for some other people beyond those who ask the questions So the first question was a decade of management meeting a couple weeks ago Maybe three weeks ago. I was one of the experts in the meet the quantitative expert session But people would come and go and ask questions from people who are supposed to know things So normally people come and ask the questions like why does not my structure like as a model converts Why do I get this weird result from an actual analysis model? How do I deal with endogeneity in this kind of situation? But this time the first question that I got was not not specific, but it was about my opinion On what I think is the biggest problem in research methods and their application in management research and I have lots of opinions about that But I was thinking why would someone care about opinions? But but nevertheless, I find it flattering that someone cares about my opinions and I have an answer to that and the biggest problem is that researchers do things That they don't really understand. So do we apply methods and particularly analysis techniques That we have really no clue on what they're based on And there is a lot of evidence that that's actually the case You can see that When you take a general article that applies something that is more complicated than regression analysis or structural So model which pretty mainstream in some areas over the last little while something more more extreme more more exotic How that is often justified is that This is what other people use when they apply this kind of problem So instead of trying to understand things We try to mimic what public papers do And that's of course a good strategy if you only aim for publication But if you really want to get results That are are robust and that are in some sense correct Then that's not ideal. Ideally you would give a methodological justification of what the things that you do, which of course Means that you need to understand Understand the analysis technique So this is in in my opinion A big problem and I'm In the process of writing an article about this. I call this a methodological crisis And we go through our papers published in journal for persons management where I was methods editor for one term and We find that maybe one third of the articles published in that journal In in the time frame that we took with us between like 2015 2018 or something like that Maybe a third of those articles contain analysis errors That invalidate the results. So for example, uh There were uh, one article says that they use seemingly unreversed analysis Because they wanted to model our two-way paths in the model. Well S u r can do that. So if you take a look at the textbook any econometrics book tells you that That's beyond the scope of they are the method And if you want to Do reasonable relationships, then you need to apply three states least squares or something like that Another favorite example of mine is an article that I reviewed that explained How they applied two states least squares by taking the residual and and using that Uh in place of the endogenous explanatory variable. Well, that's not how two states least squares works And then then I went and I checked the uh, the cited reference. So that was citing uh an article published In the same journal by some other people and that was citing an article In smj that contained the same incorrect explanation And then that was citing an article in strategic Organization that says that that article does not that there's two states least squares So we have this problem of of people using things that they're not fully qualified to do There is lots of evidence that this actually happens if you know how to look for it So, how do we we solve that problem? Well, ideally we would are all the experts in things that we do But to really like understand how you do our structure because we're modeling Uh properly and do other diagnostics takes a lot of training And maybe you don't have the interest to do that training or maybe you have other commitments So you have no time to take the training. So, uh, What do you do? My solution to this problem is to apply simpler methods So, uh, I've been telling for about maybe two years now That if you have a choice of a method that is 99% correct But you're only 50% are like are likely to use that method correctly And there's another method which is Only 85% correct, but you're 99% sure to apply it correctly always go for a simpler method because The the risk of me supplying the complex method is a lot greater the outcomes are the outcome of Misapplying that is a lot worse Than applying the slightly sub-optional technique correctly So this is the biggest problem people don't understand what they're doing And the solution is to Learn more, but that's not realistic And the second solution is is to keep it simple If you use analysis technique Don't try to explain how it works. There is another a recent example from a postgraduate student that I have And we are working on a paper on meta-analytic structure because modeling And there's this technique called two-stage scm and How meta-analytic scm normally works is that you take multiple studies your aggregate correlations into one large correlation matrix And then you apply scm into that technique in the matrix, but the two-stage approach combines All the steps of meta-analysis into a single analysis, so you don't actually construct that large correlation matrix So so but he was really confused about it And I was like, what's what's the problem? You don't just don't do the correlations matrix And uh, then he pointed out to an article in negative management journal that writes about uh These two-stage least squares and how they are using that technique first aggregate a correlation matrix and then do an scm But that's that's not nearly close what the technique does. So so this is uh, this is another problem is that You might be applying the technique correctly, but your explanation is incorrect. So don't try to explain to people how uh statistical techniques or another techniques work Because then if it's published in AMJ or smj or journal for previous management Then people will think that that's the correct explanation Just say that uh, you use two-stage least squares using status IV a regress command Then uh, if someone wants to understand what that actually does they can take a look at the IV a regress user manual Documentation in the state of user manual, which is great by the way So that's the first question the biggest problem And the second question relates to uh, this is an email that I got From a student who was asking asking for grades Which are late for my quantitative research methods course And I passed course and then he was asking me How do I learn more about research methods and uh to give a bit of context of this question This is uh, this person David He has done a my post-graduate level introductory course, which is One semester course. So that's five credits Well, it's probably more like 10 or 15 credits of work, but it's normally five credits And after that he took my advanced course, which is one year of course of of research methods training. So he had done Year and a half of training with me and then he tells me How do I learn more and uh, this is something that I haven't really thought about that much because normally when Someone asks me how do you learn methods? It's a beginner and I'll tell them to to come to my course Or I'll tell them to read a couple of good books But if you have a person who's already done like 10 books and taken all the courses the university offers I want to learn more Then my question really is that uh What specifically do you want to learn because uh, uh, once you know the basics Then it's not obvious What is the next thing that you should learn for example, if you don't know recursion analysis And if you don't know factor analysis, you don't know anything about measurement And you want to do a survey study then the obvious answer is that you go for a course that covers these basic techniques But once you go beyond the basics, there are so many different things that you can go about you can um expand to and uh, I'll be thinking about uh How to answer this question and uh, I would ask the question back is that what do you want to learn? And I have some potential answers to that question what the person might want to learn After they're already pretty good at methods and then I have some answers to these these different things So the first thing that I have in my mind is uh, that you might be want to learn more techniques But then the question becomes why is that do you just want to uh to say at least in your cv that you know 50 different analysis techniques unless you want to be uh Want to be a teacher who has to teach all this stuff like I do Or you want to be like an all-round methods reviewer which I've done then uh If if you're not in any idea of those two categories, then what would be the point in knowing lots of stuff that you don't use in your own research? And the the problem is that like if I know Let's say I I know 50 different analysis techniques Something like that. So so would I ever write article using all those 50 technique? No, some of those are are something that I would never do so so I do I do survey best. I do database uh studies using archival data But some of these things like like experimental research that I don't do I would like to do but it's very difficult to do in entrepreneurship so uh Research designs for analyzing experimental data and analysis techniques for experimental data Is not that relevant for me as far as my own research goes So so uh, uh, if you want to learn more techniques, then just think like what is the point? I would uh instead Think about learning more techniques as required and this comes to my second point. So how do you are? The second thing that you might want to learn Is how to learn? To learn statistical analysis techniques and research that has more quick. So, uh, so how do you are? How do you improve your skills so that you can just take a paper from let's say, uh, a technical journal like structure Because of modeling or psychometric and just read that understand it in one go and then go and apply that to your own research And to do that I see some value in learning more techniques And and this is one of the things that I haven't actually told my students who took my advanced course We cover a broad range of techniques and some of those techniques are They're used in management research But they're probably not something that the students would ever use The point is not to give them like a huge toolbox of lots of things But it's rather to teach them many things so that they become more efficient at learning New methods because that's what they will need to do if they want to be On the edge of of doing empirical research So, uh, the second skill that you might want to learn Is is how do you learn new analysis techniques more effectively and more efficiently? It is very easy to just read one of these guideline types articles Or take a four dummies kind of book and then go and apply without understanding Well, how do you really learn how to understand analysis techniques are Really on a fun on a more foundational level and also do that efficiently so that it doesn't take years of studying Well, I would say that there are two two or three things that help one is To read good books on on our fund events like I just got this book from um An editor who's an editor of paper of mine. So we got a revise and resolve it and then the editor sent us a Comment that we didn't understand and then I like emailed the editor asking him that could you give us something that we can study to to learn more about this issue and then He sent me this book And the book is about How you tie with tie those are things that we learn in a regression analysis class And and glm class and the instrument of variables in econometrics. How do we tie those to the moran literature on basing analysis? and causal structural models or dhs Direct data cyclic graphs that's Julia Perl's work So, so how do we how do we tie those and that is very useful because it gives me kind of like this? It doesn't give me any new techniques I know the techniques that the book talks about but it gives a different perspective like how they can just justify how they can be justified from the basing perspective and how they can justify It just if I from the structural causal models perspective And learning about needs news perspectives is one thing. So so take a good book uh Julia Perl's book on why is uh book of why is something that I would also recommend So that's not not how you do things, but it's more about justification or things that we do Another thing that you can take is just to uh to pick up like an econometrics book And it's going to be hard to read if you're not trained in econometrics But it will uh going through it little by little just take take a habit of reading it maybe two hours per week And that will teach you how to read equations And then that will help you to read on some of the more technical research papers So so read books about fundamentals Read more technical material and then the third thing that I would like to are to to You to take away is is there a usefulness of being able to simulate your own data sets So whenever I need to learn a new technique one example that comes to my mind is These techniques for endogeneity that don't use instruments So they don't use the external instruments. They're called frugal approaches to uh the instrumental variable But for example, they use the uh headers get as the city in the data as an instrument So it's very hard to get your uh head around the idea that headers get as the city might somehow be Beneficial in dealing with endogeneity because they seem kind of like things that are not directly related I took the paper and then I simulated the data set in stata And then I tried to get the analysis to work simply using two stairs least squares And I got consistent estimates under in uh endogeneity and I was pretty happy about it because I understood what the technique was about Then there are also other techniques that are still kind of like on my my My list of things to learn and one of those is generalized method of moments I kind of understand the idea and I I know the application pretty well But uh, that is something that I still whenever I have to have free time I'll try to to work through that myself simulate the data set and apply generalized Method of moments just by using matrices instead of using the canned command that stata provides So that is uh, I'm not third thing. I learn how to simulate and um then um If you want to learn uh, a deeper understanding of the math or deeper understanding of how the analysis techniques work Then uh, there's there's a bit of a problem because human learning how it works Is that uh, you need to challenge yourself but Not too much. So there are the same thing. Uh, I I can give you an uh an example for weight training So, uh, if you want to learn how to uh to bench press 200 kilos, which is something that It requires lots of practice years You don't you don't go to the gym and then try lifting 200 kilos at the at the same time like right away And uh, you go to the gym. Maybe you bench press 50 kilos when you start And uh, you don't go and you you bench press 50 kilos year over year and hope that you get to 200 What's the trick is to are to choose choose something that you can barely do and then do that And then choose something that is uh slightly more difficult. So uh, like you would do bench pressing You would start with 50 once you have our been able to do uh five rex of 50 You go to 55 and then once you do five rex of 55 you increase again to 60 So there are different programs on how you do that and after a few years you eat the 200 if you eat Well, the same thing applies to to learning research methods. So, uh When I started learning uh methods, I remember that I was the kind of a person who just took a software Then I read out for Dami's kind of book and I was I managed to do something with the software and uh, I did some regressions And then I moved to structurally because of marvel because I was able to Publish conference paper with Richardson and I was then why not go with scm because that's what they are All the cool guys are using and then then I did something using lisrel And I got a conference paper feedback. They're either told that the author clearly does not know what they're doing And yeah, that was the case. I I was following this kind of like cookbook approach without really understanding what I was doing And uh, the model that I I did did not make any sense if you understood what the model was about So, so how did I then then proceed? I I started reading A regresson book by cohen cohen et al 2003 and that contained equations and uh, that was really hard So so I had not read equations. I I I'm normally pretty good at math So, uh, in high school I was pretty good at math. I did some math during the university But I never liked statistics. So I could make the like two or three From the probability and statistics course. So that's not my thing And I'm trying to get my head around the equations was was difficult And then what I did was start reading something else So I read about applications that come they maybe with less equations and then I started to eventually are develop my understanding of equations so that I could then take cohen's book and read it comfortably and um Then I realized that maybe this is not the best book The same thing happened with with uh, wool with this introductory econometrics. It was Or uh, to to start with it was really hard I had to struggle through the first couple of chapters that contain equations And then then I saw those equations being applied elsewhere So I read for example, uh, paper novels and service those methods that applied are the same kind of equations slightly different context and then then that developed my understanding and my ability to read equations Then I came back to war. It's another real estate. Well, this is uh, this is simple thing And now I'm getting to the level where I might start Reading uh, some part of Bullridge's proper econometrics book the Longitudinal and uh time series data something like that and uh To read that pretty comfortable not all parts of it, but but most parts So so you kind of build gradually your understanding. So pick something that you are That is slightly Challenging for you but not too challenging and then just work through it Maybe set aside two hours per week on on reading that thing And uh, then another thing that you can you can learn is uh, is is this these fundamentals like if you if you have a Regerson book it tells that Regerson analysis has been proven to be consistent and biased And some of these these books contain proves And the proof might involve a normal distribution of their term The the question then that you might have is How would ever anyone come up with that kind of proof? and I'm not reading a book about one mathematical proof. It's it's more for fun It's it's called thermats last theorem. It's not about Statistics or analysis, but it's just how how mathematicians come up with ideas And uh, you might might start to think about Uh, how did anyone ever come up with revision analysis? How do we prove regression analysis? unbiasedness and What is the normal distribution? How do we derive normal distribution? You can do that, but do it out of your interest. So so I don't know Uh Where normal distribution how who proved it and and how it's it's properties are derived And if someone gives me a proof that regression analysis is is consistent and is unbiased I can read the proof pretty comfortably But if someone gives me, uh A clean slate of paper and asks me to prove that regression analysis is consistent I'll just give the paper back to them because that is something that I can't do And you don't need to so so it's it's then it clearly becomes like like, uh, do you want to understand it? Uh out of curiosity understanding Where it proves and come from is to probably not very useful for becoming a better researcher And I talked with uh with a former colleague Henry Stewart about this, uh The same issue except in the context of philosophy I had read some philosophy And I was excited and I thought Henry that this is actually very useful And then Henry applied back to me that you haven't read enough. So, uh, once you are You read certain point then the things that you learn about might be really interesting There might be intellectually challenging to you But ultimately what you learn from these sources Does not make you a better reader a better researcher In the way that you can actually actually apply them in your research practice Then, uh, final thing that you might might want to learn is, uh To use your software better. So, uh Statistical software can be used in multiple different ways. I got some screencasts about it where I compare different uses and, uh The way we normally start using software if we ask Stata or SPSS, which is not really a professional stool in my opinion However, many people are Really productive with SPSS, but then data analysis is probably not their strong thing Uh, so, so how do you uh, why would you want to learn more with the software? The reason is is reproducibility and efficiency Like, uh, we have a paper that is now invited for third round in general applied psychology And we run like a hundred or so structural equation models that are our variants of the same basic model And uh, one co-author did the analysis And uh, he did uh With the models one by one, uh, specified them using m plus then run it save it specify another model run and save it And that that is a problematic practice for for two reasons. One, it is error prone So if you point and click or if you type Then, uh, the risks that you do something, uh, that is different between the models is real and secondly takes a lot of time So learn how to automate things and how to do a little bit of programming with your statistical software And uh, this is uh, something that you can you can learn by by doing you can just like If if you decide that you want to learn how to do reproducible tables using stata you would pick up the status collections and tables manual and then Work through that and learn But the real question here is is how do you force yourself to do it because it's a bit tedious to learn about these techniques even though it pays off in the long run And my my recommendation on that is that that uh make rules for yourself. So make a rule that you will uh, You will never type a single, uh, regression table In your life and then every regression table regardless of how complex it is Must be produced by exporting from the software And when you set these challenges to you then it makes uh, it increases your skill Then you can move on to have different kinds of tables to do with that software so so Build these habits of of doing things in software that you could quickly do manually and ultimately your Uh level of efficiency with the software increases. So that it becomes actually faster to do it with software instead of doing it manually so I have these these four things that you might learn more techniques or designs and uh This you would I would recommend that you do this only because uh learning more techniques helps you to learn to learn And uh, then but there's a small caveat here If you want to be something, uh, I come up with clever analysis designs and research designs So that you can be someone probably see something in your field as the first person ever Uh, I have a one tip for those people that I got from John Antonakis And uh, he comes up with these clever research designs, uh particle experimental designs And I asked him, what does he come up with all the ideas of doing this stuff? And he told me that he reads Science and nature. So he takes a look at what they do in the real sciences and then, uh, he reads those In his bed before going to bed and then uh, he gets ideas By by looking at The other fields that are even not related to us. So if you do an experiment, uh, how would you apply that same design? to your own discipline And a deeper understanding of math The same applies to deeper understanding of Of the actual estimation method. So when you are estimate a model maximum likelihood on a computer Then that is numerical optimization. You can you can do a pc degree on numerical optimization. It will not make you a better researcher So, uh, once you understand it on a certain level, like you understand what is a hessian matrix What is a gradient vector and how they relate to the estimation then understanding the optimization algorithms Beyond that probably not very useful If you want to do it as a hobby if you like doing it, it's intellectually challenging then go for it But don't think that that is uh, something that uh makes you a better researcher Uh, and then the final thing was the use of software just force yourself that there must be a way of doing this so with software like I have a Two figures that I want to uh Want to combine when I'm using state. I I published the paper earlier this year about Uh model similar about transformations and non-linear models eight eight guidelines something like that and we talk a lot about visualization And what was a really learning experience for me was to do some visualization with data first And then try to reproduce the exact same visualization with r because the building visualizations. They look don't look the same And sometimes in stata you have a building visualization that plots one curve And then I wanted to plot another curve there So of course, I could have just done two curves with transparent backgrounds and put them on top of one another Uh using uh Some graphics software. No one would have noticed but then again Pushing myself to to do it all in stata and all in r instead of of doing anything by hand Help me to become more proficient user of this software. You can take a look at the r-code and state according to appendix of the article it is Open access a link uh the article to the video description So you can you can get it from there and it's pretty advanced r and state according to some cool tricks that I learned during that so uh then uh Sounds a practical things that you can do to improve yourself and one thing that I haven't really uh Touch on yet Is that how do you how do you learn about the current practices in your figure? So uh the social science uh management within the social sciences a social process in in the sense that uh would be Think is is publishable and and good enough rigorous enough It is kind of like a contract between researchers. So there the resources. So there's no like like Reason to say that let's say P is uh more than 0.5 is is not Not a result P is less than 0.05 is a result. So that's just kind of like an agreement or standard The same thing. Uh, we have a great management that if you don't know, uh, what is the functional form of between x and y Then uh a line with a line will do if if you are no No reasons to believe otherwise So there are these these agreements and these standards within your field and it varies between different fields I recently talked a couple months ago. I talked about talked with an epidemiologist and uh Their standard for causal claims was that if uh two things are correlated Then we can make a claim that there's causality And then it's up to others to disprove it and that wouldn't fly in management research But it flew it is how some people publish in high-profile journals like like nature So uh to understand the conventions and understand that the problems of of methods in your field uh I would I would say that becoming a metal tree viewer in a good journal is one of the best things that you can do So so I run the method reading process for general person management for about three years And I did most of the papers myself, but I also train some students using those papers and uh, I know some of those methods editors and uh I I know that there's a shortage of people who are skilled on methods who want to uh to review Papers for their methodological correctness and completeness And if you volunteer and you can make a credible case that you actually know what you're talking about Then uh, that is going to be very attractive for for these journals like any any good journal So I I know that's uh, for example, uh A leadership quarterly has a pool of methods reviewers Uh, the journal of management has a pool of method reviewers general authors management building one Entrepreneurial theory and practice used to have some kind of method review at least so so these these are journals have pools of reviewers So if you want to get involved You can just email an editor and ask and then but that requires to actually put up what you do and then that will make you even better and uh Yeah, and then finally, what do I do to to learn a method? Well, if I really want to learn something like uh, how do you do uh lagged variables? This is something that I learned last year I uh agreed to do a course where I teach lagged variables and then that gives me a deadline for learning how to uh How to model these dynamic models dynamic panel models how to do uh, how to interpret results when lagged variables are present and uh that when you are able to uh To teach something explain something to somebody else Then uh that helps your own learning. So even if you're not uh, like running a course for doctoral students, uh You can you can now have a colleague. So if if one of these uh, if you're learning a new method, uh, for a paper That you're writing with a company with a set of co-authors Tell the co-authors that you run half a day workshop about that method for them And when you start thinking how do I explain things to others? Then that really uh Uncovers your hidden assumptions. So you might assume that you know something But when you start thinking about how do I explain this this thing to somebody else Then that really reveals if you have really understood it And in a way that is is fundamental enough that you really Are able to teach So so these are the things uh that that I would recommend So so as as as just just go and and study new things decide what you want to learn Uh, do methods reviews Try to teach some principles to others like if you're a third year doctoral student Run a class on first year students. This is actually something that I started with So I was a doctoral student and I wanted to teach my colleagues how to do SCM Of course, I did not not teach it the way I would do teach it now and I would probably are My my current myself would probably tell my my past myself from 10 years ago or so that That you don't know what you're talking about. So that is One thing read widely not only your own field, but other fields of entry applications Read research methods journals like august as a resource methods is a it's a great resource for learning What new things are available and also There are articles that point out some perhaps problems and misapplications of of the current things that we use And then you can also learn about those so In a nutshell these two questions that I had where What is the biggest problem the biggest problem is That people are using things that are not qualified to uh to use or they don't understand How they work or what are some sort of they're based on and then the reviewers are equally concerned about these methods Often not always but often and the second question is that how do you learn more about methods when your basic understanding is already Pretty robust. So so well simulate learn challenge re-challenging stuff Become method reviewer and then start reading these more applied Method journals like organizes or research methods and then Psychological methods sociological methods and research. These are probably the three most relevant ones for applied researcher in management disciplines so, uh, thanks for watching and We have the green screen up. We have uh, i'm a wife. We're both recording stuff for for courses and then it will be uh, there will be some some more, uh, maybe 15 20 Sort lectures coming on coming up in the near future