 Our next speaker will be Frank Bosco talking about Metabus, a platform for managing 1 million applied psychology findings. Welcome, Frank. Thank you very much. Good afternoon, everyone. I'm very honored to see so many. Chose my talk over a happy hour. Thank you for that. I'm going to talk today about a research curation project from the field of applied psychology. I'm going to show you some of the under the hood. I'm going to show you some demonstrations. I'm going to talk about some past projects and current projects. And maybe we'll have some fun. I need some audience participation. So be on your toes, please. Anyone who's ever conducted a meta-analysis knows that it's arduous to collect findings from primary studies. I think a typical meta-analysis takes between one to two years to complete. I have an example here from a psychological bulletin paper from 2001 by Tim Judge, where they claim to have manually searched 21 journals over so many years. And if you do the math and assume all the journals go all the way back to the start year, it winds up coming to nearly 11,000 articles that were hand searched, which is a monument of inefficiency. And let's contrast that with, say, crockpots. We can find just about any information we want about crockpots with a finger snap. We can filter by size, or what have you, and ascertain information about crockpots quite quickly. So the need for research creation, in my view, is becoming quite clear. We're starting to see some efforts in psychology. Actually, I shouldn't say starting, it's been years. Neurosynthesis, an interesting case. But we're starting to see these pop up. And metabus is the one, I suppose, in applied psychology. I should note, though, applied psychology is quite diverse for those who know the field. It's not just work variables. It's going to include things like personality, like well-being, demographic variables. So you can ascertain quite a bit of information about the human condition from applied psychology, as from others. Let me jump in and let me show you a typical abstract from applied psychology. This is from the Journal of Applied Psych. I've underlined five constructs that are mentioned in the abstract. For those iopsychologists in the room, you might be thinking right now, this. This is the correlation matrix. That's what you would expect to be in that paper. This is great because we have every correlation between every variable in the study, and we can use that information in meta-analyses if we want. But that's actually what was published in that paper. There's much, much more information in these papers and an incredible repository of findings. And so what we decided to do was to build procedures for extracting the information in the matrix, which is semi-automatic. And then we have a manual coding process by which we classify each of those variable rows. I'm highlighting two right now because I don't have time for all of them. But for example, we collect information on the data collection country, and we collect information on a taxonomic code representing the variable type, which I'll show you in just a second. And in fact, I'm gonna jump in right now. This is where I need your help. Any applied psychologists in the room, I need you to shout out, if you would, common variables or constructs that are measured in applied psychology, not topics, not like employee selection, but a particular personality variable, a particular behavior, a particular attitude. Let me have a few of them. And I'm gonna, conscientiousness, okay? Let me have another. Where are you married? Okay, let me have another one. Did I get conscientious? That's kind of the same thing, isn't it? Let me have another one. Job satisfaction. Job satisfaction. So conscientiousness and job satisfaction I can do. Okay, so I'm gonna run a meta-analysis right now. Don't judge me, but I happen to know the five digit code for job satisfaction is 20072, but I'm also gonna type, my doctoral students have to know them all. There's 5,000, now I'm kidding. Conscientiousness, I don't remember. Conscientiousness. All right, I'm gonna specify job satisfaction using the taxonomic code, but also the tax, and I'm gonna specify conscientiousness using just letter strings. This is gonna ping Amazon web services, look for those findings, come back and do an instant meta-analysis for us on the fly. We found 117 findings whose mean R was a 0.16, and you can actually scroll down and see all the information for each one. And God willing, you can click a button and through the magic of the DOI go to the individual paper and see the information about it if you'd like. But there's tons of information in this database. We have, like I said, there was a country of origin, I have sample type employee versus student and various other things. Response rates, the alpha value, if it's reported, et cetera. We can also do what I call an exploratory meta-analysis. Somebody mentioned job satisfaction. I'm doing all meta-analyses with job satisfaction right now, and I'm only showing meta-analytic summaries for which there's at least 300 effects. Or I can drill that down or I could show me all meta-analytic summaries with at least 200 effects. And the color represents the absolutized and then mean correlation, and the size of the circle represents the frequency of study. So very quickly you can see across all of these meta-analyses involving job satisfaction, which tend to be most common and which tend to provide the strongest effects. For those who know the field, Frank Schmidt with colleagues, Huili, I think, wrote a paper on the construct redundancy of effective commitment and job satisfaction, which in my opinion are pretty much the same thing too. And if you look at the entire plot, it's the reddest or darkest of all the bubbles. So maybe this is some evidence for the construct redundancy. I would buy it for sure. I'd love to show you more, but I don't have time, but well, maybe I'll show you one more thing. And through the magic of our shiny, this is all built in shiny, you can do fun things where you can build the chloropleth maps, for example. So these were all those findings that are the 114 or whatever it was. About half of them came from the US or a little more than half came from the US and so on and so forth. I'm gonna go back to my slides. I only have five minutes here. Okay, I'm gonna show you quickly one study that was recently published using this database because it's not just for creating a platform that lets people search. People are using this for meta-analyses across applied psychology and me and my team do all kinds of fun things with it. This is a paper where we took them, we have by the way, a million findings or so, 1.1 million findings in Metabuz and for almost all of them, we have the country of origin. And so we said, why don't we test to see how much does culture matter? How much does culture, whether operationalized as Hoff said or Schwartz value survey or what have you, how much does it actually explain variants and bivariate relation magnitudes, right? So we actually did 136 separate meta-analyses where each one is a popular topic such as conscientiousness and job satisfaction. I would not be surprised if that's among the 136. And we were able to estimate that across all of these different operationalizations of culture, culture explains roughly five to 7% of the variance in findings. And I thought this is fast in editing because you really can't do an analysis like this without mega data as it were, without a whole lot of information. It's difficult to do an analysis like this. And Metabuz is used regularly to so-called fill holes in meta-analytic correlation matrices and also to do entire meta-analyses. The study before this, I'm sorry, no, it was just after this. We published a paper in plus one on p-hacking. And one of the problems with p-hacking research was that in journals very often people just report the p less than 0.05 or p less than 0.01. So you never really have the exact p-value reported. Well, Metabuz has the sample size and the effect size so we can calculate almost exact p-values for a million findings. And then we can nuance them by the bivariate relation types. So for example, attitudes to behaviors, personality characteristics to attitudes and so on and so forth. And we did find slight evidence for p-hacking but it wasn't anything to cry about. But again, I really do think data at large scale and curated data are needed to do analyses like this. But I think the most exciting paper, the one I wanna talk about just briefly here, I mentioned before construct redundancy, it's an enormous problem in my opinion in applied psych. I'm not sure if it's as bad in psychology proper. But we realized that we have all these data and we decided to build an enormous, I know you can't see any of those numbers, but picture this as a correlation matrix of essentially all the major variables we study in applied psychology. Through the magic of R and various other analyses, we can actually do a factor analysis of the entire field. And smaller scale examples are available. This was published in Nature Communications on the topic of self-regulation. The title I think is quite telling, you get the basic idea, uncovering the structure of self-regulation, the construct under study, through data-driven ontology discovery, which is really what we're trying to apply to our entire field's database. Problem is there's some holes in the matrix. If anyone has a solution, please let me know if you're inclined. Another potential way of analyzing it is with network analysis and using what they call community detection. Not quite an expert on it yet, but I'm currently learning it. But again, starting point, unless you have a starting point of a massive corpus of data, I don't think any of these approaches would really work. I want to highlight just one quick thing before I end. A lot of the exciting network analysis stuff that's going on in meta science right now, I think the UCSD map of science, relies on co-citation information. Co-citation information is indeed very cool, but it doesn't tell you about how strongly related things are. It's me-search, right? Or perhaps somewhat closely related to me-search, who studies similar to others, right? I'm always reminded of Frank Schmidt's comment that the effect size is what science is all about. We curate effect sizes at MetaBus, and we do fund analyses with them. And thanks for your attention. We have time for a couple of questions, if anyone has any. Hey, so I think we've heard, and lots of people know about kind of the issues in psychology with oversampling in weird countries. I'm wondering if in this particular data set, you also may be concerned about the fact that the same individuals might be sampled multiple times. Like how many times is the average psychology undergraduate going to be measured unconsciousness across multiple different papers and therefore show up in your data set multiple times? So a violation of independence? Yeah. To be honest, I haven't thought about that. I suppose it's possible. My guess is it's probably a small concern. But let me add one thing to the, I just completed a study, or working on a study with another colleague, about, I think we're targeting the Journal of International Business Studies again, about the frequency of data collection across countries, because we have this information. The logic, and this might be a surprise to some of you, but the notion that our data in psychology are weird and biased and overly American is officially false. That line was crossed, I want to say, in 2008. At which point, maybe a little later, maybe 2012. At that point, U.S. data in our corpus declined below 50%. And it looks like within the next five years, there will be other countries that will overtake the U.S. as the sole largest providers of data. So, yeah, I'm kind of answering two questions here. I've never thought about the independence assumption. Honestly, my guess is it's probably not that big of a concern. But yeah, it could potentially impact something. Great. Thank you for the update about weird. 2008 is when I left psychology. Okay. I'm not up to date myself. Okay, thank you for your question. Thank you, Frank.