 Welcome everyone who's here. I'm Kathy Zeiler. I'm on the faculty at Boston University School of Law, and I'm a board member, a current board member of AMOS, which is the organization that is co-sponsoring this webinar with the Center for Open Science. AMOS stands for Association of Interdisciplinary Meta Research and Open Science, and I wanted to make a quick plug before Chris gets started for the annual conference. This year it's going to be held in Brisbane, so if you're looking to have a nice vacation in Australia, it's going to be on November 21st through 23rd. Please see our website for details and the registration should be up soon. I went last year. It was absolutely fabulous. It's two days of really great talks. But we're going to attend to the Meta Science Conference now, and I'm very pleased to be moderating this panel, this webinar, as we lead up to the in-person conference in Washington, D.C. next month. And I'm very happy to introduce Chris Koch. He is going to talk to us today about a metastatic approach to examining the structure of implicit memories. And I'm going to stop showing my screen so that Chris can share his and then take it away. Chris. All right. So I should also say, sorry, Chris, before I hand it over. For those of you who have questions or you want to post comments, et cetera, please use the Q&A button, not the chat button, but the Q&A button. And raise your hand, and you can either speak your question. I can elevate you to speak your question to Chris. Or you can type it into the Q&A, and I can read it to Chris. Whatever you prefer, raise your hand if you want to jump in on the screen. That's fine. And then you can have a dialogue with Chris if that's going to be easier. Either way, you can do that throughout. So Chris invites questions and comments as he is speaking. So I'm going to monitor those and make sure they get in. Okay. Take it away, Chris. All right. Well, I'm going to talk to you about some metascience related things that we somewhat uncovered as we're working with a larger task. Looking at implicit association tests. So I just want to do a few things before we really get started. These are some of the students who have been working on different aspects of the overall project and have contributed to today's talk in a variety of ways. So metascience, I probably don't have to really go over this very much for this particular conference, right? But the goals are really to help identify basic elements of scientific inquiry, right? To figure out how these elements might interact. We're looking at how we're going to determine scientific knowledge, how it's produced. But the last one is really where I bolded a few items because that's what we're going to probably focus most on today is really improving the quality of the research that's being done and also reducing inefficiencies. And if we look at the Venn diagram over here, we're really looking at mostly at data reuse and maybe a little bit on statistical critique. But those are probably the areas that will hit most throughout this. So just a little bit of background. This is going maybe a little bit more than what we need for metascience, but it gives you a reason why we're actually looking at any of those to begin with. When we look at memory, memory is generally broken up into explicit memory, things that we're consciously aware of, and then implicit memory or the things that we are not consciously aware of. Semantic and episodic memory. So memory for facts and events and events are typically semantic and episodic. And then you have a variety of different types of implicit memory. And the one that probably is most tied to the implicit association test is that of priming. And you can see priming it. We have changes in perception and belief caused by previous experience. And so that's, that's what we'll probably focusing most on just within implicit memory. But there's also how to knowledge and perceptual learning and other types of associative learning. Now the implicit association test has five blocks of trials, and they're all structured basically the same way. You start off with some concept words. So that's in the first block and, and you have keys usually on two different sides of the keyboard. So I was just for to demonstrate for the talk today, I'll just mention a and L. And so you may have a concept word related to individuals weight. Right. So you may have, you may have like muscular or you may have heavy or something like that. So these are concept words around weight. And then if it's, if it's a positive trait, let's say we'll press a, if it's a negative trait, we'll press L. Okay, so that's your concept, those concept words and then you have a value of words and the value of words maybe like, you know, just basically just like good, bad, right, lazy, right, whatever, whatever words. So that with a particular I concept you're looking at. So we may have words that are related to positive aspects of individuals and then also maybe some negative things, and negative things probably might be related to the stereotype that potentially out there. So they're going to press a and L for those words as well. And then they have a third block in which they get both. So now you press a, if it's normal weight and good, you press L if it's heavy or bad. So you have the combination of those underlined, underlined congruent here because those are kind of congruent with the stereotype that we might have like a person who's very heavy might be lazy. So those two things will go together. And then, and then what happens is you switch the words so you switch the keys that they press. So now when it's instead of lazy and they're hitting L they press a key. So you switch the coding of the concept words, and then you have them do it again. So now the concept word for lazy is with someone who's a normal weight, instead of someone who's overweight, which is more stereotypical. And so those then would be the incongruent pairs that you would have, and you're really interested in the difference between these two conditions between the congruent condition and the incongruent condition. If something matches our belief or implicit belief system or bias, we're going to respond faster than if it doesn't. So that's, that's really what you're looking at with this IT. And IT has been around for quite a while, which is part of the reason why we looked at it. It's been researched fairly well for about 25 years. So here are some typical words in IT just to give you an idea what the evaluative words might be like. So you have words like happy peace joy right on the good side and terrible war agony could be on the bad side. I bolded war just because the analysis I'm going to show you in a moment has to do with word count. And this is just matching word war or just matching the word in this case war. So it doesn't look at variance. So it doesn't include like worship or plain war lord war torn pre war post war right all those are in a in a corpus but they're not the words that were matched. So here's, here's what you find with this and this is, I took the labels off because it was a little bit harder to read but you're looking at frequency on the y and response time on the x. And so it happens, as your frequency goes up, your response time is slower responding to the words as a frequency goes down and takes you longer to respond. This is kind of typical of a frequency that we find with explicit memory. The other thing is this is barring response times from another study so the one nice thing in terms of like that. This conference is that there are some larger data sets that are out there that allow you to get some of this information without doing the study yourself in a sense so you can really get a good idea of what's going on with. In this case the words that are being used, and the good words are responding to actually faster than the bad words, just by responding to the words so this is not including them in the it yet. This is just if the words are on their own, you respond fast to the good words and the bad words. Right, and you respond fast to the words that are more frequent than less frequent. This is looking at the good versus bad words are used actually in the race 18 to see a similar pattern. So the good words are respond to faster than bad and you have a frequency of bacteria as well, where are the more frequent with the word the less the slower the response, I mean the faster the response time. So, when I just kind of like take those kind of try to account for those word frequencies from existing information. You can actually see a decrease in the t value that you get for the study for the comparing the congruent incongruent condition, and there's a decrease in effect size and that's probably not too much to triple a thing because one of the criticisms of the it is that the effect size are not always very large to start with. And so, when you account for the word frequency it seems to even decrease some more. So if you do this overall correction, there is some indication that you really are getting some frequency impact on it itself. And so that's an important one important reason for looking at the frequency and the it. The other thing is this, and this is part of the reason why we started looking at this, and we don't have to worry too much about the rest of this diagram. This is part right here the semantic system. So when we talk about frequency effect I mentioned that it's usually associated with explicit memory. So if we go back to that slide, right explicit memory we're dealing with semantic memory and episodic memory and we're oftentimes probably even more focusing on semantic memory, which is general knowledge. So, what's different here though is we're using an implicit task, and those implicit memory tasks are, as kind of suggested right here right these are indirect measures of something which is why the it is actually a really good task, because we're just looking at a race it that you say, you know, you can ask somebody or you racist and they're probably not going to say well yeah I am thank you for asking a minute. They're going to cover that up right so we have self presentation skills that we engage in when we know that someone's asking this question that would be inappropriate for us to respond in a particular way. And we want to present ourselves in a more positive light than maybe we actually think. The implicit association test it doesn't allow you to do that because these are immediate responses that you have their actor, they're operating below your level of awareness right below consciousness. And so you don't have the ability to control in the same way as you would if you're filling out a questionnaire. So, the thing that be interesting and part of the reason why we started taking this on is that, if there's a frequency effect with these implicit tasks. It could very well mean that we're asked we're, we're taking memory, right and we're we're accessing memory the same with the same type of mechanism it's just a level of awareness. It's a process that changes between explicit and implicit, and a reason here's some different ways you can you can kind of conceptualize that right so explicit memory is going to be more of here a conscious mind, you know, interfacing here with thoughts and feelings. And then you have insights intuitions things like that these are kind of more pre conscious. And this way, it's kind of hasn't to show this because this is usually attributed to like trying to describe Freud's theory but but you know I feel like an iceberg above the water this is all conscious mind right below is this pre conscious but then you have this big part of the iceberg is really much lower and that's that's the unconscious mind. And what we're really talking about is, is that if we kind of think of this as memory. We're accessing memory, but some of some of the time we're aware of it and some of the times, we're not. If we're aware of it's explicit or not. It's implicit, but we're getting the information the same way if we show these similar effects across implicit and explicit types of memory tasks. And that's a way to kind of look at why this might be an important thing to examine, because this has those different types of memory that I've been talking about, but also associated with the different parts of the brain. And so usually when we're talking about things and we think well semantic memory is really more the medial temporal lobe, whereas priming is going to be more like neocortex. And I was thinking that okay since they're operating in different parts of the brain they're actually different things. Right. And so what kind of suggesting with this frequency effect. Within within an implicit memory task is that suggesting that we're actually have a memory store that we're possibly accessing either consciously or unconsciously. That's that's a big reason for the motive behind the study itself. Now if we it's relatively easy to do to look at the frequency effect, and that all we all we have to do is kind of recode the data so that you know when we know what word is presented or what images we can see if it's a higher low frequency, we can actually put in the frequencies from like a corpus or something like that, and coberry them or we could categorize them as high low medium frequencies and then we can analyze to see if there's a difference between those. In order to do that we actually have to have trial by trial data and so that's one of the things we'll get to but there are some other reasons for picking the it for this and that is, like I said it's been around for a while. And there's also the project implicit website through Harvard, and they have a lot of people going there and doing these tasks. And so they literally have millions of people that have done various types of it tasks, and that data is largely available. And you can look at things over time so I'm going to give you an example of looking at something over time and one of the, one of the things that you know we've kind of started exploring just by looking at these differences over time. Now, what they have found Charles was in the nausea found is that there are certain biases that seem to have decreased over time. And there's certain biases that seem to have increased so the one example that we've had on the slides as a race it. That seems to be decreasing. The other one I mentioned the body weight that one seems to be increasing along with age and disability. So there are changes that can happen with the implicit associations that we think people are making. And this is an example of one of those things when trying to figure out why. Okay, so we can look at these over time and again so this is kind of you can see visually what these tasks are like so here's your concept. And the word in this case it's an image so you have to say whether this this individuals are European American African American, then you have your evaluative words have to say whether this word is good or bad, and then you have both together. Right. And so that's your combination. This is from a different site I mentioned implicit project implicit site. This is the OPL stands for online psychology lab. And this is done through a PA or the American Psychological Association. And this is looking at data actual response time data from individuals from 2006 on. And you can see that the incongruent times are slower than the congruent times, pretty much across all that time. Right. And that's what you would expect based on what I've been saying about those associations are making this though is interesting right because you have this big bump in the response times right and, and the difference between the congruent incongruent trials, and not only is it a big bump or the times are much slower and the differences increased, but it's also opposite of all the other times. So, seem to warrant some further investigation. Right. And, and just to point out right so this is primarily when you look at OPL site you're looking primarily at college students, some community college students and sometimes are some high school students, but they're probably junior or senior so that's kind of the age range you're looking at, maybe to 22 something like that. And, and we took samples we looked at samples from primarily introductory courses and from various regions around the country so we're both the country up and took different samples from different regions and we use both public and private institutions. And when you do that you see something like this here are the times. Or here are the differences that they have and you can see this is actually starting in 2010. So in 2010 11 and 12 they're up higher like they were before, and then you can see that they drop. Okay, this is actually looking at each individual school as a sample, and you can see the difference across the different different samples or classes across time. And it really is those 1011 12 times that are giving you a slightly different level of it than more recent times. So the question then is if this really is something that's out there why. So we start looking at a few different things and this is a great way to look at some other tools are available that look at large amounts of data to try to help us figure out what's going on in our in our studies. This, this top graph is really just from, it's from psych info, actually. And so looked at the number of publications, regarding it per year. And you can see that there was a steady increase I said, it's 25 years so the original article is 1998. Looking at it, and you can see that steadily increased to about 2013 where peak is kind of dropped off. This is Google searches. So using Google trends looking at Google searches for both either it or implicit association tests. They, they, they're right on top of each other so they're pretty similar. This peaks the timeline so much the same this peaks in 2008. So it seems like they're the public interest in it kind of about five years before it did for researchers. So, that's, in a sense, interesting but part of the reason we looked at this was, well, was there an increase in interest in the it around 101112 and did that promote some type of difference in how people responded. And the answer that is probably not because the peak increase is much is earlier than the change we saw in that data. But this is looking at just people just looking at race. And you can see this is 2010 starting here. And there's no difference in people how people search for race before or after that time period where there's that change. If this bottom line is, you know, maybe they're looking at other things or besides just race. This is a critical race theory and that's where you see it's just flat. There's no differences there. Is it due to some high profile event that happened that you had a lot of media coverage. And we started looking at some of the earliest, you know, really big media events around. In this case, violence. And those all occurred after right so those didn't seem to be good. Hate crimes actually seem to decrease during the time period where this increase in the it happens so that doesn't seem to be a great explanation. So when you look through databases that had the tops news stories for every year. Nothing in particular really stood out. Looked at economic factors in 2009. The wealth gap had increased to its largest amount at that time. That directly preceded when there was this bump so is that possibility for why we saw that bump and possibly. And when you look at additional data, there was also an increase around 2021 and there wasn't an additional bump in that it curve we saw. So that doesn't seem like a great explanation. So let me start looking at other things right could it be a could it be a political figure right someone who's popular in this case. President Obama was elected and was elected in 2008 inaugurate 2009. The change came in 1011 and 12 so these are shortly after he assumed his presidency. Now the interesting thing in terms of this being an explanation is that we didn't see the bump start during the campaign, and we didn't see it continue throughout his presidency. So if this is a possibility only lasted for a relatively short amount of time. Right. So what's that cause for that spike in implicit bias. That's an interesting question. It doesn't seem to be a fluke in the data because there's the same task. And we had a consistent response from students from around the country. There was a period of time, the various reasons that I just talked about they all are possibilities but they don't seem particularly satisfying. And it may be that we're not looking for a a factor but it's, and it makes sense if it was a combination of factors actually that all kind of came together at a particular time that changed the associations that people were making. And if that's the case, though it does suggest that there's some economic political social sociocultural factors that can have some type of temporary effect, at least on the associations with implicit bias that people have. Now that if you can figure out exactly what that is. Well now you have some ability to possibly influence negative biases that people may have. Now that may have some additional ethical concerns associated with that, but that's those are those are some possible implications. So the it again is great because there's, it's been around for a while you have a site that collects a large amount of data. And then, you know, we can see there's actually multiple sites have a large amount of data on it, and we can look at trends and can, you know, download the data and we can examine it. The part is all great. But what we needed to do for ours is not just look at the data that they had, but we needed to look at the, the trial by trial data mentioned that before. The trial by trial data tells us how they respond to the positive word or the negative word, or the, the word that, you know, like if you if you're looking at depression, for instance, which is an it. Sad is used much more frequently than something like melancholy. Right. And so, if you're if you're looking at those we need to know how do they respond to each word. Every time those words are presented. And that's what you don't get when you go to these data sites, because they have the basically the clean data right the data that is in that final analysis. There are some summaries for each person, but you don't have their individual responses across all their trials. And for an it that can be, you know, we're just giving an idea, or talking about an action I'm kind of jumping ahead of slide but this part I just want to say you know there's there's different things that we decide about data and one is like what data do we include. This is just from a study looking at inequalities and education. And you can see they're starting off with, you know, just over 485,000 people that seem to fit the general requirements. But as they start looking at more and more of their requirements related to the study, they end up with just over 24,000 people that that they can look at. Right so and then, depending if you're doing other research you may say okay I'm only going to include people that respond at least 75% correct right there if they're making more errors that they're not going to be included in the data analysis. We can look at people that have an ordinarily long response times and say okay anybody who's responding more longer than this, which is really beyond the what's typical, then what they're not going to select them. There might be people who you know that might skip a question or two and there are techniques to fill in missing data but at some point there's too much data to fill in right so we have to say if they if they don't complete a certain amount, then we're not going to include them. Right, so there are ways that we have their criteria that we set as cut off values say this is data we're going to keep this is data we're not going to keep. And then we do that analysis will come up with a data file, and that data file that we do the analysis on is the one that we usually post in these data repositories. And that's the kind of data that was available for the it. There's the other thing about related to this right is the data maintenance part which is, it's not just the data that we collect right it's also the data that we make available and the data that we keep. So for instance, I can tell you that the OPL site when they did the it, they, they automatically calculate the congruent and incongruent times, and that's the only thing they say. So you can't go back and get the trial by trial data for that site because they made a choice in the beginning what would be saved and what would not with a project implicit site we can actually go back and get original data. So you're looking at a huge amount of data. So, I was looking at the race it just for 2022 there over 750,000 people who've taken that. That's a lot of participants. And then when you look at their individual files individual files are probably going to be about 120 trials. So in each trial you're going to record. You can you can vary how much you record but you're at least going to need like what condition are they in. What was what was presented what word or what picture was presented, how long it took them to respond what response they made, and whether or not the response was correct. So you're looking at probably at least five pieces of information for each trial for every participant. So when you start adding it up that's a lot of data, right, and try going through individual files for like 750,000 people I mean that's, that's just a lot. So this is actually from an article he was talking about data storage and not research itself, but is a copy because it was, it was seemed relevant here. So some things that we do we want to have it be cost efficient that's one very practical thing. But then, how well can we access the information. Right. And when you start having so much data, it's almost becomes overwhelming as to what to do. And so that's an issue and then also how much space do you need what's what's the storage requirements, and how easy is it for you to distribute these data files. So these are all all important considerations. And you notice what as he was talking about data storage, his algorithm is basically you pick two. Right. So we may get something that's really cost efficient, and it's really flexible and easy for people to access information. But it means that it might not. It might be reduced information. Right. And so we're making, if we pick to the other one might be compromised. And, and so we I think we kind of see that a little bit here, but also it also makes sense, right. These are just practical issues we have to deal with when we're dealing with especially large amounts of data. So here's here's before I go to the next slide. So what I've really focused on mostly here is is the words. Okay. So the words we can look at we can get a corpus we can look at we can actually go to Google we have an n gram viewer we can get word frequencies are pretty easy to get, and we can control for frequency. Right. But there's another aspect to some of the it is and that is it's not just the word that's presented. It's also a picture that's presented. And so that's what we can we're looking at here. Now this is great again because project implicit they also have an associated OSF site. And on that site you can download the data from 2000. I think it's 2000 might be earlier in 2010, but you can get data for every year. And you can also get these these are their picture files that they showed. Now, one of the things that is important when you're looking at faces in particular there are certain dimensions that are that help us recognize who people are. And there's certain features that are kind of less important. What I did is I took we took these these pictures right these 12 pictures and and borrowed some questions that was used in another study. And so we took these pictures individually put them up on the screen fall to the design of another study where they showed a picture, the picture went away, and then they asked a couple they asked a series of questions. So here are the questions right how blank was the previous face. And so on a scale one to seven or you have dominant threatening right trustworthy, all those things what we're really interested in this particular case. Are these two things that they included how stereotypically black is the face how stereotypically white is the face, because how that face matches up with a stereotype can influence how people respond. So here's one set of questions, here's the other set of questions. You know how likely you sit next to the person on the boss or sharing or things like that. So these are kind of more face, more behaviors related to the individual, but this is this is kind of what their initial perceptions are about that particular face. Now there are other things that we could ask. And some are kind of more, you know, maybe kind of more biometric measures like what's the shape of the eye, how big are the eyes how far apart of the eyes how white is no is how long as a nose. You know, we can ask questions like that those are all important for face perception as well. Symmetry is important. And that seems to be tied to attractiveness and attracting this can influence responses depending from what we're asking. So there are a lot of things that we can ask about a face to see how good a face is to use in a particular study and bring it up because we're going to come back to that in just a moment. But in this case, looked at these, all these faces based on those questions and this is looking at them specifically for how stereotypically white or black that people perceive them. And this was done using put the face and put the start the questionnaire into we posted through m Turk and had a couple hundred responses. And so all these women are rated as more stereotypically white than black. All of these women are more stereotypically black than white. And all these men are rated as more stereotypically white than black. I move these to the side because these faces were not rated that way. So there's no difference between how stereotypically black or white they saw these faces. And what's significant potentially significant about this is that they're all the male faces, all the black male faces. This is not something that's kind of spread out across all the conditions of the face. This is one can one set of faces that seems to be responded to differently than the other three sets of faces, which may introduce some systematic bias into the responses. It's important when we're using different different types of stimuli that we understand exactly what we're showing and what we're asking people to respond to. Now, in order to do that we kind of need like a stimulus repository where we've gotten much better at data repositories. But what about repositories for the stimuli themselves. We do have some good places to go to, but they're not really stimulus related. So there's, there's this one there's psych test so if you want to find a test, and I didn't add this bold it was right there right instantly find and download instruments for research and we're teaching right so you can go here and you can get a test. And if you're interested in some type of personality dimension, you know, other some other type of individual difference that you can look at, maybe a particular condition. There are a variety of things you can get a measure for and evaluate where does each individual that have in my sample fall. There are other sites like this, you have like the positive psychology site the University of Pennsylvania, they have a variety of measures that are used for like authentic happiness and mindfulness and things like that. And not only do you have the names of these but you have the description of them, you have links to them if they're available. You have the original sources that for the references and all the psychometric information so they're great places to go to to get a lot of information about instruments that you can use. There's an IP IP site international personality item pool, you can go there and there's a lot of information about different sets of questions that you can use for different types of things that you want to look at really to personality, and it has again all the reliability that it has links to original articles. Again, another good place to go to but they're not like faces or voice files or object files or things like that that we might use another types of research. And so that kind of gets us to this idea of, well, do we need to start developing stimulus repositories in a more strategic way. So, here are some of the desirable characteristics of data repositories this is from NIH. And so you can see a variety of those. And most of these are really applicable to a stimulus repository as well. So, I won't go through all of them right but you know, persistent identifier so being able to create a DOI for these, having them being available for a period of time, but then also this metadata. And not just if we go back to the phase example, not just the face here's a set of phases that I use in the study, but all the information that goes along with that face so, you know, before we did it we tested it and here are all those, like those mathematical measures that I said. Here are some ratings that people had about how attractive they thought the person was, or how, you know, stereotypically whatever they thought the person was. Right so you can have all that information about every face that's included in this in this stimulus set. And then as a researcher I can go there and say, when I want to do a face perception study, someone has basically done all the background work on those faces for me. And that's, it's I can reference it. Alright, so the people contributing to it have reference and people that are using it have the reference as well. But that metadata is going to be really important to know how well to know what what's appropriate use for those. And then also, like the curation and quality assurance. So those are going to be some of the big ones obviously some of these others are going to be important as well. But this is, these are some practical things to take into account when we're looking at maybe creating a stimulus set. Now, these sets can go here. So dimension in a more systematic way. And so what might be great to do is to actually have specialists in these areas so you may have specialists associated with face perception stick with that one. And they can get together and say that these are all the important dimensions that we need to evaluate for a face to know if it's good to use in a study. And then those become the dimensions that you collect data on those become the things that you fill in for your metadata with associate with each picture. Right. And so now you have a set of experts are saying this is what needs to be done to have a really good quality stimulus and then and all that information is available for all the pictures. And it could be faces. I mentioned voice files. If you're doing like, oh, you can do like emotional processing of voices or something like that. Well, what makes a good voice for a particular emotion. So those are kinds of things you can use if you're doing object perception research when you're looking at objects and you're seeing what people what people can how they identify them. Right. So, you know, just you can look at a number of different areas of psychology, which was where I'm focusing. And then and then build out these data set stimulus sets based on the recommendations of experts in the area. So you basically have like recreating a set based on best practices. This is what needs to be included or considered when you're using whatever type of stimulus. And I think this is kind of I've heard this. I've heard this people say this. And it's kind of interesting to talk about like in about this at a conference. Right. But this is you're dealing with a bunch of people from a bunch of different areas right so a lot of you are not going to be psychologists like I am. Right. But we're all together, right, dealing with meta science. Okay. Now, so what happens like in psychology, what's happened in psychology is that these large conferences that used to everyone we get together. They've even become specialized. And so what will happen is, you know, like I'll go to a conference and it's going to be mostly like perception people. And the social psychologists who might be interested in prejudices, prejudicial and stereotypical behavior. They're not going to be at that conference anymore. Right. So we're missing the interaction between different disciplines that might have been or different subgroups within a discipline that might have been helpful in in developing, you know, really well and, you know, tightly done experiments, because we don't interact with each other anymore. And this kind of tears that down because it provides a space where all that can take place and we can benefit from the expertise of people in other areas, as it pertains to the studies that we want to do. And that's kind of what we're thinking as we're talking about. Not data repositories, but stimulus repositories to try to help improve research. And so here are some of the things just kind of getting down to a summary here is that the IT is great in that there are some things that it does really well. And I didn't mention this earlier, but you can go on a GitHub, you can download an IT example, where in theory all you have to do is change your stimulus files, and then everyone can be running the exact same study. So if we can all agree on this is the best methodology, you can download something like that and we're all doing the same thing, which is obviously going to help turn the visibility as we include all these different samples that we're working with. So the data and the stimuli are all available in OSF. Right, so that's great so we can go there we can get all that information. So some things are really good about the IT. But when we start looking at specific things like does frequency of the different stimuli matter, or does how stereotypical face use or an image used in one of the other ITs. We can't get that information because that information is not saved. So we may have to look at something like do we need to include trial by trial. And the one that I the one that I mentioned with over 750,000. So what we may need to do in that case is say, look, you just have too many. That's too much data. So if you have a situation like that, maybe take a random, a stratified random sample of all your participants and you, and we make these repositories, you know this type of data available in repositories, like up to this many participants. So we're going with maybe 300, 300 is a lot less than 750,000 but it's a lot easier to deal with something 300 or less than something that large with the IT. So, so there is some need for additional information that is collected that we don't all get. And so sometimes these things sort of make doing other follow up research a lot easier. And that kind of thing about it. If we're able to show a frequency effect with someone else's data who didn't design the study to look at that. Right then, neither of us are potentially biasing our study at all right because they're they originally didn't collect the data to look at it, and we didn't collect the original data right so the some of that experiment or bias is going to be eliminated. So we have a developing standardized stimulus sets that are a variety of sets that are available for researchers to use to really make sure that they don't have any potential confounds in their study, and that they can be assured that they're using the best possible stimuli for the studies themselves. So, those are kind of the recommendations we had going through this process. I mentioned some of these already in terms of what a stimulus data set should include, and what days data repositories to potentially include as well. And I believe, yes it is that got me that's that. That is everything I wanted to say about this based on the project we were engaged in, and it does leave us some time so if you do have questions I'd be certainly happy to try to answer those as best I can. So if you have a question if you want to jump into the queue please raise your hand and I'll see it pop up I can click on a button that will allow you to speak at your question, or you can just write it into the, the q amp a. Um, so we do have one question in the q amp a from Judy, Evara Sue, and Judy says, does it, does it mean that associations and implicit memory can change due to extrinsic factor social events political events etc apart from inherent individual differences. I think it's possible. Yeah, I mean I think it's possible especially like if you, if you think back to the one slide I had with priming and it had based on experience, all those social things that are happening are actually part of your experience as well so it's very possible that it can. Now to the extent that it changes it over a period of time. I can you can't tell from the data that I showed you how long the impact is going to be. It looks like it might be temporary. But I would suspect depending upon the type of experience that will be longer lasting. Great. I have a, I have a question. So, I completely agree with your recommendations I think they're, it sounds like they're, they're really valuable. You know the more data we have, for example the better we can, we can attempt to replicate work and minimize differences between the original study and the replication attempt and the other positive benefits that you suggested are great. One question that I've been thinking a lot about is kind of institutionally how do we create incentives for people to post these sorts of that information on, you know, OSF or their data repository to think about those pieces as data kind of how do we, how do we get this going other than coming to conferences and presenting the ideas and hoping that they'll the people listening will catch on and actually do it. As a good, it's a good question and part of it. Maybe, I mean the journals are probably going to, they can play a part right. So some journals now won't even review unless you have posted your data somewhere right. So that that's a possibility. The other thing is, you know, if you spend some of these things creating a stimulus set that's really good can actually be quite time consuming, but it hasn't really been considered scholarship. It's gets you to the ability to perform the scholarship that you want to engage in. And so maybe reevaluating or, you know, somehow giving different different level of credit to developing high quality data stimulus sets would be a way to encourage it as well. So if you can actually create something posted and that becomes a valuable reference for you for your research other people's research. You know, because these these. I don't think I mentioned it specifically but you know I mentioned about using experts as a way to evaluate like what things should be included, but you can have experts evaluate what's in what's actually uploaded. And so if you include like a peer review process to that, then it becomes something that might be deemed much more, you know, source of scholarship and would encourage people to do as well. So I think there was probably those two things would have the would kickstart at the most. Yeah, that's great that the Open Science Foundation actually had in their guidelines has one of the recommendations to formalize citation practices around data set so that people actually have an incentive to create data sets that other people are using and they get cited for them, which, you know, is our kind of currency of credit. Do you do has a follow up question. How do we select concept words for studying personality concepts when they're extrinsic influencers as well how do we partition between inherent influencers and influence due to extrinsic factors. So, so the concept words. Yeah. I think I had a part of that one slide that said it's, I think these, these words are really tied to stereotypes. And so if you're looking at, I mean the concept words, right for sure. If you're looking at weight, you're going to look at normal weight and overweight because that's that's where the bias normally is with people that are heavier. And then what are the things that people think about naturally when they think about people that are overweight. I don't know if that's a great. There's almost like a circular definition in a sense is, you know, what are people normally associated with heavy people and then then we put an implicit memory task and then we, and then we get it, we get a different way. Right. But you're looking at things that are tied to really more of a stereotype of what they think. Now it doesn't always have to be about a stereotype as well. Sometimes the stereotypes are not as clear. There are certain things that people just like more than other things. So there's one of the first it was with flowers and insects. And so you had a different set yet a set of flowers and then you said okay this is a flower this is not right. This is an insect and you just you just categorize that way. And then you have your value towards and people rate the flowers as being not, you know, better than the insects that people generally don't like insects the bad words get associated with the insects. Good words go associate with the flowers. And so in that case, you have like here. What are the typical flowers that people usually have, you know, are aware of. And so you want to include those if you start using names for flowers and no one knows what they are than the name and I even know it's a flower. And then for insects so it kind of depends on the concept some of these are, like I said are kind of stereotypical types of words or words associated with a stereotype and others are kind of like I'm looking at flowers and looking at insects. I'm going to pick common flowers and common insects. So it kind of depends on the concept I hope that answers that. Great we've got another question from Emma Mills. She says, did you say at the beginning that you may touch on statistical analysis critique so could you talk a little bit more about that. Were you referring to researcher degrees of freedom and analysis choices here. Yeah, you know, I was, I was thinking if you go specifically for like the example that I use about work frequency. That is something we can statistically control for that was not done in the original. We can either go in and say, oh, they use this word. Here's the corpus I'm using here's word frequency, and then we can covariate, or we can categorize things by like low high medium frequency words, and look at it that way. But that was really kind of going back and saying okay if we if we want to take a task like the it and look at a different variable that they didn't include but we can, we can reasonably include based on the data based on what should be in the data files. Then we can add that variable in, or we can recode something and then we can redo the analysis accounting for that other factor that we think might be important. So that that's really what I was I was thinking about in terms of, you know, something was missing. And the data sets allows to go back and add that in. After the fact, then we can actually make the we can make the correction off the original data, instead of having to do the whole thing over again. Another question from degrees of freedom yeah there's a lot of choices researchers get to make and, and when you're doing something that's really large and get a tremendous sample size, you know you're getting these really nice p values but sometimes not very high effect sizes. Yeah. Good we've got another question from our boundary, a lot to be this person asks stimulus there's it's a comment actually, and we want your thoughts on this stimulus availability over time may lead to familiarity which could potentially influence the responses to them. Yeah, that's true. And so, you know, you may need to exclude people who have participated in a similar state before that cuts down on your potential to collect data so maybe that's not optimal all the time. That would probably be something to look at and, and also with the other part though is, is that, you know, these, ideally, these would be like dynamic stimulus sets that you're developing and so, you know, you may have a group of images for instance that you're looking at but then people add to them over time, and then you have a pool that you can pull from instead of just looking at a set but but yeah you probably you may want to consider asking a question upfront about whether or not they've seen any of those items before or something and then they'd be excluded. Yeah, so there's there's a good part in the bad part right influences the research in a negative way that's obviously not good but if it also increases our ability to use better stimuli and generalized findings better than that's a that's a good part so there might be a little bit of a trade off and an appointment which you know we have to control for that. So just in our last couple minutes here wrapping up and anyone else can jump in. The meta research community has been looking at it pretty intensely and I think one thing that people seem to be talking about is whether it's possible to separate implicit bias from explicit bias you know how do we know where that these implicit association tests actually measure implicit as opposed to explicit bias. Any thoughts on not more general question. Yeah, that does. That is one of the things that people talk about like what exactly is it measuring. I think I mentioned kind of in the beginning, ideally. People are making associations that they're just making these quickly near courage to go as fast as you can right but you also don't want to make. You don't also don't want to make mistakes mistakes are based on how they, how the pairings are expected to be. So if you don't respond with whatever with a good trait with with a race it mean it's kind of set up as bias so that you have a positive bias toward the white faces and so if you said the other thing then it's considered wrong. Yeah, the. It is harder. I really do think it's harder to consciously control what's going on with an it just because of the speed at which you're responding. So is it is it really tapping more implicit associations it probably is. You could, in theory, I think probably slowed, slowed down just enough that you're now consciously. You're consciously controlling your responses. That's possible and but not slowing down so much that your response times look really off. So, and then if you go back to what I was talking about with memory I think. You know it kind of goes along the same path is what I'm what I'm saying really is that you have you have your memory. And you're tapping into that memory. One you're doing it you're doing it without a lot of awareness and when you're doing it with more awareness. And so it's the same if we're dealing with the same source of memory. And if we're dealing with the same amount of awareness, we may be able to move on that's on that continuum of awareness, but we're still accessing the same thing. So, yeah, I mean what what exactly is the measuring can you consciously control it. I mean, it's possible. I want to squeeze in one more of the people that want to get in there. There is great capacity for planned missing data design here so that stimuli are not overexposed. What do you think about that missing data design. Yeah, so like exactly what what are you referring to for the missing data design part. I'm just going to click on a lot of talk so we can hear your voice if you want to jump in. You have to unmute if you want to be heard. Can you hear me now. Yes, we can yeah. Thank you, Chris for a fantastic talk. And so I'm having the same kind of thoughts with cycle linguistic research across like 29 variables and differing effects in the literature, and a standardized stimulus set, like the question would possibly become overexposed over time. So a way to mitigate that might be to have them in different kinds of bins if you like, and when people take them they take them in random selections. And over time, we piece together the data so that not everybody sees every stimulus but we're collecting enough where we can do missing data imputation. And so it also works with if you're using very expensive and stimuli, you give that to a core set of your participants, and then you give the cheaper stuff. So it's something that I'm thinking about. So that when you get your experts together and they do that huge standardized yeah what do we really need. You can use it. And because everybody I'm guessing everybody would want to use it to begin with, but you could kind of ration it. And there is a kind of theory around it or a method around it that I've heard about. What do you think. No, I think that sounds like a good idea. And, you know, like I said, if it's dynamic that the more it grows the easier it is to do I think what you're talking about, and prevent the overexposure that's kind of come up. But yeah, I think that that'd be that'd be a good way to do it. And you could, you know, you could probably, you know, you can manage this as well. I mean it's possible. So, instead of having, I mean, you wanted to be freely accessible, but you also may want to say well how many of these are going to use and which, you know, so we may have some level of awareness of how many times in particular. Well, you actually can do it by downloads. Right. So even if you do it randomly, you can know how many of the individual stimuli have been downloaded. And then you can make sure that those are not if they're frequently coming up that you spread them out. I mean, I don't know if I'm saying that particularly well, but, but yeah, I mean, I think I think the more you have the easier to do stuff like that and that'd be that'd be great. That'd be really good for different areas of science. So I'm glad you mentioned something totally different than what I mentioned. I think I think the curation of it becomes that scholarship idea that you were talking about as well. Yeah. Yeah, I think so. Really interesting talk. Thank you. Thanks. Alright, so thank you everyone for attending. We hope to see you at other webinars. Check the meta science 2023 website for additional webinars coming up. Thank you so much Chris. Really interesting. Take care everyone.