 Hello, everyone. Thank you for coming. My name is Monica Gonzalez-Marquez and I just realized I forgot to put my name up here. I knew that I was forgetting something. I'm heading and have designed this workshop in combination with various people. Ines Schmall, Francis Castanso, Jeff Beye, and Allie. I just completely forgot her last name because that happens to me sometimes with names. So we've prepared this hoping that you find it useful in understanding research that you read and if you're in particular, if you're attempting to reproduce a study, that it becomes easier for you to understand what you're supposed to do, whether it's you who's missing something or more likely that there actually is a great deal of information missing in the documentation that you have. So reading and comprehending is very much an active process. So let's go on ahead and begin there. The first thing that I want you to do is to look at this image and tell me what you see. That is, yes, just tell me what you see. And Crystal, can we allow people to speak? Yeah, everyone should be able to unmute and talk. Yes. So please do that. Tell me what you see. Okay, can you say that again? Your voice is coming out really garbled. Yeah, it's a bird that sort of like got dragged into the snow, I guess. Okay, so you're seeing a bird that touched on the snow. Okay, good. Okay. What else do you see? Other people? I'll go. I have no idea, but I'll say I have no idea. Other than it looks like some sort of dragging print and the snow that somehow diverges. Okay, there's some tracks and just know the diverge. Yes. Okay, good. What else do you see? So right now we have some kind of a bird and we also have tracks. What else? I think it looks a little like an arrow. It looks like an arrow. Okay, cool. So some kind of metaphoricity here with the image. Okay. Or well, not metaphor, but okay. Yeah, some kind of an arrow. What else do we see? I'm also a big fan of the Socratic method. So if you don't reply, I will pick on you. So for example, there are also trees and shadows back there. Yes, there are trees and shadows. What time of day is this? Oh, afternoon. It's in the afternoon. Why do you think it's afternoon? Oh, it seems a little dark maybe. I don't know. Okay, okay. And then you've got the shadows. I suppose if we had a larger picture, we'd have a better sense of direction of where the shadows are coming from. Okay. What do you think the tracks belong to? Josie? People? People? Why do you think they belong to people? I consider it. It seems like there's something like a drawing around it. So it might have been done by people somehow, not by animals. I don't recognize which animal could have done it. Okay, okay. So you're referring to the tracks in the snow, right? Yeah. Okay. So you're seeing bird imprints, you're seeing tracks, you're saying there are trees. Where are we? That's a park, potentially. Yeah. Yeah. Okay. So what you're seeing is an image of a bird of prey leaving an imprint in snow after it has captured its prey. In this case, a hare, which is a large rabbit for people who don't know that word. So we're seeing the imprint that's left. The image has a rather dark title to it. It's called Lunch or Bunny Lost. Now I'm going to ask you one more time. What do you see? But this time, I'm going to tell you. What you actually see is a collection of pixels in shades of gray, black, and white. That is what is going into your brain. However, when I asked you what you saw, you didn't answer that about the physicality of the media that you were seeing. What you did is you interpreted the image. And the way that you interpreted the image was about something that happened. And what exactly happened here, what happened is that there was a bird of prey, and it was hungry, and it captured something, and it flew away. And we were able to extract that kind of complex information from a bunch of pixels in not even any real color, just shades of black and white. Now why is this relevant? Because it says a lot about the way that human cognition is set up. It is set up for narrative. Now in the title, I clarified that we were not talking about psychom. And I will develop the clarification very briefly. What psychom does is it takes a narrative in a scientific paper, interprets it, and retells the narrative in such a way that the science communicator believes will make that narrative easier to understand. So it's actually a superimposed narrative. What we're going to be talking about today is not a superimposed narrative, we're going to be talking about the narrative that is inherent to the scientific process, that is very much what structures the process, and that therefore becomes very obvious in scientific papers once you learn to see them as narrative. Now we're going to do a couple more exercises just to drive home the idea that yes, science is a narrative, it is a sequence of events. And then we'll continue with exercises that will help us understand the information that is missing in papers by using narrative structuring and the tools that it provides. So now let's continue, you're going to see this slide. So there's going to be links to materials, they're going to be in the chat, and otherwise it will be on the screen. But what I'm going to do next is I'm actually going to turn off my camera. Okay, and I am going to show you something. Let me get my props here and make sure that everything is here. So stop share. Okay. And why do I not see, okay, do you see me? Yes. Yes. Okay, perfect. Okay. So the first thing that we looked at was a two-dimensional image, a picture. Okay. So now tell me what this object is. Can you see it or is the camera too high? Camera's a little too high. There it is. Okay. So I got my little pedestal to make sure to see if it was a little higher, but I guess I shifted the camera. What is this? Coffee cup. It's a coffee cup, right? Why do you think it's a coffee cup? It looks like one and it's morning, so it makes sense that it wouldn't hold coffee. It's morning for you, it's afternoon for me, but that's okay. But what makes it look like a coffee cup? The shape, the familiarity. Familiarity. Now that's an interesting word, okay? So it's shape and what is it that's familiar about it or what exactly makes you think that you would do things with it that are coffee cup-like? Yeah. You know, when you go into a coffee shop and you order your coffee to go, it generally, you get used to it coming in a cup that looks like that. Exactly. Exactly. I know all of this seems really obvious, but trust me, it's not all that obvious. You'll understand in a second. Okay. So now tell me what this is. A pen holder. Sorry. A pen holder. Now it's a pen holder. What changed? You put some pens and pencils in it? Mm-hmm. So I engaged in a different behavior with the same object. Okay. So what is the point of this little exercise? I asked you how we know what objects are. So how do we know what objects are? I'd like to say we make deductions, but also we assume. Yeah, we make assumptions and we make deductions, but what just happened right now? Word starts with the you, use. I was going to say context. There's more to it than context, and it's also more specific than that. It is actual use. Objects are how we use them. I'm not saying that that's all there is to them, because what else do we need of this object in order for us to be able to use that as either a coffee cup or as a pencil holder? It needs to have a specific shape, right? It needs to have a specific shape and it needs to have certain affordances. Now I don't know how many psychologists are here or people who've studied human cognition, but objects have affordances, and all that really means is that their shape is such that it allows us to do certain things, but that isn't enough. So there are theories in psychology that argue that the shape of an object is enough for categorization. We know it isn't because it depends on how we actually use it. The reason that this matters is because every single time we use an object in a certain way is a tiny event, is a tiny story. So this is another example of our using basic narrative structure to understand the world. It occurs everywhere in everything that we do and try to understand. I just got a notification that this was starting and it's 15 minutes late. Anyway, so this happens everywhere in everything that we do as we categorize our environment, as we learn to understand what our environment is. Now where else does narrative occur? I just said everywhere. Let's talk about a social sphere. What's an example in the social sphere where we exchange narratives? Sorry, could you repeat the question? Sure, of course. What's an example in a social sphere, in a social environment, in a communicative environment that we are exchanging narratives or exchanging stories? I use the words interchangeably. I suppose it might be a situation like this, for example. Exactly, how is this a narrative? I suppose you're sort of trying to lead us through a story of a set of ideas. A story of how our story works. It's a meta-story, but it's still a story. What are some other examples from everyday life? When you tell your partner about your day at the end of the day? Yep, that's another common example of stories. What about when you're sharing things with your partner that happen that have none of your business? What do we call those kinds of stories? Gossiping. Yes, yes. You may or may not be familiar with Robin Dunbar, but that researcher proposes that one of the biggest motivators for the development of language is actually gossip. Our need to know stuff, our need to get stories about stuff that is none of our business, but about which we are still intensely interested. So this is a narrative guiding so much of our lives. We pick up a newspaper, we are reading stories. The structure that you see there is not an accident. It is very purposeful. Journalism discovered a long time ago that to keep people interested in stuff that happens, you need to present it to them in ways that humans understand information, which is stories. This is precisely why newspaper articles are written the way they are, to present narratives as directly and as clearly as possible. Now, when we look at a scientific journal article, I think that often we forget or actually, no, let me rephrase that. We don't realize that what we're reading is a narrative, that it's actually an intentional narrative. When the scientific article was first designed, the idea was specifically to document, to better document the narrative encompassing what people did. This is why the early papers were actually letters. Because when you're writing a letter, you're telling people what happened. So it was very clear at that point that what people were exchanging were narratives. Then, letter writing became cumbersome for a series of reasons, not least of which was the need to disseminate the information about what you did as a researcher with a broader audience. Hence, the paper was born. The reason that it's structured exactly the same way it is is because it is a narrative. So let's talk about the paper as a narrative. Let's look at it as a narrative. You'll see that it's actually amazingly obvious. Does the introduction tell us? Debbie, what does the introduction of a paper tell us? I guess the introduction starts off and explains the whole work in a way that makes it make sense in context to things that have come beforehand. The keyword there is beforehand. What happened before? When we're reading a novel, the first chapter does what? What does the first chapter of a novel do? Sets the stage for what's going to be coming. Exactly. Does the introduction do exactly the same thing? It's a story, plain and simple. Now, the one thing that I will clarify is, and I will use a very mild expletive, but it really is suiting here, it's really important to distinguish between narrative as inherent to the scientific process and making shit up, which is what people usually think in science when they think about storytelling. We are not making shit up. We are once again directly dealing with the narrative inherent in the scientific process. That other kind of storytelling we have no business with and we don't care about that. That's what the introduction does. Once we set up what the reason is and what happened before, what would happen next in the story? Where is this found in the paper? Larry? Larry is not here? Andrea, what happens next in the story after we've set the stage? Oh, you're here. Did everyone just go away? I'll jump in. Literature review. Okay. And what is the literature review tell us? It's the previous stories that were told, what people have done on related topics previously. Exactly. It's the stuff that other people did. Okay. It's the stuff that other people did. And then after that, and that is still where in the paper, what section is the literature review usually found in? I'd say the literature review is found in the introduction. So the next section is the methods, but I don't see how methods translate directly to a in a nonfiction story. We're not interested in whether it's fiction or nonfiction, but just whether it's a narrative. Yeah. And so how does methods translate to in a story? Okay. What is the methods? What happens in the methods? Well, you're telling what happened, what you did. Exactly. Any section that is very inherently narrative-y, it is the methods. That is a description of everything that you did, of everything that happened. So at the beginning, yes, in many ways, it's the plot. Yes, exactly. I mean, it is the plot. So at the beginning, you say what people did before, why they did it, etc. And then you say, and then you also describe or talk about what you're going to do. And then you describe what you actually did. Still narrative. It's all narrative. Okay. And then what happens after the methods? What's the next section? The results. Yes. And what do we describe in the results? What we found. Yeah, we describe what we found when we did what we said we did. Once again, still narrative. Okay. And then what is the last section? Analysis. And what is the analysis in the story? Why, what you found matters, what it means. Yeah, what it means. Yeah. And specifically what it means in relation to what you thought was going to happen. So very briefly, I'm going to share my screen again. And I'm going to go to the whiteboard and let me get this out of the way. I'm going to draw an arc. This is not a graph. It's an arc from narrative arc. And let me make sure that I have my tools. Why do they always put the tools in a different place? Okay, we are in whiteboard. Yes. Here they're tools. Okay. So this is a narrative arc. And you begin with beliefs at time one. And you end with beliefs, two, at time two. Okay. Here you have your background. Okay. Here you have the thing that happened that creates the possibility of a change between, actually she said belief or state A or one, two, two. That's what happens here. And then we have the conclusion, whether that happened or not. Not the concussion, but the conclusion. Okay. So this is what we have. This is exactly the same basic narrative structure of a novel. You have something, you have a set up in a series of characters and a novel, I mean, in things that are going to happen or that happened in the past. And then we have that pivotal event that changes everything. And then at the end, you decide whether you're going to change your mind about something or not. And you discuss that and you explain why this change here made this or this event here made this change possible. This is very, very basic narrative structure. Is everyone with me? I usually actually deconstruct a story like a little bit of writing hood, but we only have two hours. I was trying to do this in a bit more condensed way. But is it clear? Or are people still not believing me? If you're not, it's really important that you say so otherwise the next parts are going to be hard. We're all good. Okay, good. So then I'm going to take a quick screenshot here. So we have this and I can make it available afterwards. There we go. And I am going to stop here. And then we are going to continue with the presentation. Okay. Which is, I'm actually going to put it here because that is easier. Okay. So let's continue now. We're going to go ahead and deconstruct an article and extract all of this information that is missing or come to understand what information is missing. So the first thing we're going to do is we are going to read this article and let me close this or move it out of the way. I put it in an amber background so it's easier to read on the screen. But I'm also going to put it in the chat so that you can download it if you want. And I want you to take a couple of minutes to read it. And then we're going to talk about the general sequence of events that happened here. Okay. The end of this article is see the ball. And where is my chat again? There we are. Hold on a second. It's not letting me access my files so that I can actually give you stuff. And now there is my chat again. There we are. I'm just going to put everything there so that it doesn't, I don't have to keep doing this. Crystal, it says access to file restricted by your account admin. Interesting. Yeah. Let me look into that. Okay. That should not be that way. Okay. It may take me a few minutes to do this. Okay. Then let me just go back to the presentation then. And just looking at this text while you figure this out because we do have some exercises to do here. Is this comfortable for everyone to read on the screen? Yes, it's good. Okay. Then go on ahead and read. Okay. And just take a few minutes. Whoa. Back. I'm really sorry. I do not know what this thing is doing. I think it's on slideshow. Then it goes ahead on its own, but you might be able to just show it in full screen and then you have control over the moment. Is that it? Okay. Thank you. I'm not sure. I think maybe. Yeah. It's just switching back and forth between so many different of these viewers. It's okay. So now it's full screen and you can see it, right? That's what matters. I can see it. Okay. Perfect. Okay. What we want to extract is the general sequence of events in very general terms. Is everyone done reading? Not quite. Okay. Big question for you, Monica. At the end of the paragraph in the rightmost column, it ends with and how many, how should that sentence end? Oh, thanks for catching that. It was probably a formatting error. You see how to get this back on screen really fast? Well, I mean, you're staring at my full screen, so I guess it doesn't really matter. The Mac is complaining. There we go. Here is the paper itself. Let me zoom in. So I think you only shared your powerpoint. You're not seeing my screen? I always see part of the powerpoint. Oh, great. I am literally sharing my entire screen, not just the powerpoint. Okay. Oh, because it, okay. This is different from what I did before. I have a Zoom account and the controls are different than what's happening here. Stop share, share screen, the desktop. Now do you see my screen? Yes. Okay. Perfect. Perfect. Okay. So I think they're down here. Stop it. Nope. And it's not letting me zoom in to make that bigger for you. I thought technology was supposed to make our lives easier. Okay. I reported how many times they had been at bat, their number of hits and walks, and how many times there we are. Is that big enough? For me, yes. Thank you. Okay. Okay. So I will assume that everyone has the rest of the text now. Okay. And let me do this. Copy. That's not what we wanted. And of course it was going to do that. Okay. I have no idea how to do this quickly, but you've read the text. Yes. Anybody? Yes. Okay. Sorry. This usually does not happen. Usually I feel a lot more in control than what's happening right now. This is very, very annoying. Oh, because it picked up the rest of the thing. That's what happened. Okay. Hold on. Because I just realized we're actually going to need this in a minute. Crystal, have we figured out yet how to download, how to upload stuff? I think the alternative might have to be uploading it to OSF. Okay. Let me do that really, really fast. Here's the link to the, if that helps. Thank you. Okay. And I can just drag and drop really fast, right? You should be able to. Yeah. Okay. Great. Thank you. Oh, okay. Great. Thank you. Just create a quick folder. Oh, here. Actually, if you want to drag it to that one specifically, that one's yours. Oh, okay. Thanks. Just close on me. Great. That is the last one. Okay. Fantastic. The participant bar is just insanely big on this one. Okay. Great. And I have everything here. And now, of course, it's just totally gone. Views sort by date last opened. And I want this as, what's going on here? Oh, you need to be logged in. Oh, okay. That happens to me too. Okay. Thanks. Okay. Sign in. And it's on my, okay, like that account. Okay. Now we're in, now we're to let me. Okay. There we go. Great. We are working off of the screenshot for now. Okay. You've read that. I am going to go ahead and switch over now to our spreadsheet here. Okay. And this is, no, that's the participant one. This is the learning worksheet. Okay. The first thing we're going to do is we're going to list the chronology of events in that screenshot. All that I've done is copy the introduction and the method section. For the purposes of this workshop, we are completely ignoring the rest of it because that's what we're interested in, to see what information is missing from the method section. So this is one of the reasons why I really wanted to have you looking at my entire screen because then I can do this. My sleeping kid. Okay. So I'll assume that you're looking at the same document that I'm looking at. So many things here. Come on. Go away. There we go. Okay. Okay. Let's start by describing exactly what the events are. What happened? What is the sequence of events for the study? Nicole? Well, they start by talking about or quoting players of various sports and their perception of the ball. Okay. So they're quoting them? Oh, yeah. Okay. And what is it about? What is it that they're quoting? The ball looks bigger when they're playing well and smaller when they're not. Okay. Quoting players saying that there are differences. Can I spell? Nope. Of the ball. What happens next? They make comparisons to other sports that tennis players say the same thing. Golfers say the same thing. Basketball players say the same thing. Okay. What else? They're sort of in that section with a proposal of a generalized mechanism. It's like that. How they feel they're doing is modulating their perceptions. Okay. So they propose a mechanism. Okay. Sounds good. And what happens next? They conduct an experiment and they start by sort of recruiting some players from intramural and city softball league. Okay. Okay. So they propose an experiment and they start recruiting. Okay. And then what happens? Unformed consent. Good. And then what happens? It looks like when they were recruiting, they were recruiting at softball fields and they like advertise that they'd give them free drinks. So people who had just completed playing games were offered a drink and then asked if they'd like to participate. Okay. So what do we call this that they did? They were recruiting. It's just interesting. They talk about recruiting and then they talk about consent and then they talk about recruiting. So maybe they're still talking about the recruitment part. Okay. Okay. So you want to say this is still the recruiting part. Okay. What happens next? That second recruiting part feels like the incentivization of participation. Okay. Incentivization of participation. Okay. Then what happened? They show participants a poster with a series of circles on it and they ask the participants which size they think best corresponds to the ball. Corresponds to what they saw? I think the ball that they were playing with does that understand it? Oh, to the ball. Okay. Yes. Okay. Cool. Okay. What happened next? Ask them about their performance. Okay. SPPs about performance. So this is self-report, right? Yes. Okay. What happened next? Amy. They calculated their batting average. Okay. Who did? Okay. The researchers did based on their self-reported. Yes. Performance. Perfect. Yes. So it's the researchers who did it. Okay. What happened next? They took the demographic information about age and sex. Okay. And then whether the team won or lost, which I don't know why that would matter. Okay. Okay. Then what happened? That's it. Okay. That's all that we have. Yeah. That's all you have. Okay. Fantastic. Now the next thing we're going to do. Yeah. Let me actually set these up so that the text wraps. Tools. Format. Alignment. No. Dead one. Okay. Well, actually I shouldn't do that. Well, on this one it's okay. But if I do it in the other one, then the columns will get all wonky. You're all going to be sharing a spreadsheet in just a second. Okay. What I want us to do next is to take that same text and what we're going to be doing is extracting the verbs. We're going to find the verb. We're going to go sentence by sentence. And right now we're actually going to focus just on the method section and say what it says. So let me pull up the thing here. The text. Where did it go? It's right here. I hate this thing. Especially because I don't even get to see your faces for the most part. I just get a list of names that get in the way. Okay. Here we are. So here is our text. Let me move the spreadsheet over a little bit so we have the right column. And let's start at the beginning of the method section, which is that second paragraph. Okay. What's the verb in the first in the first sentence? Recruited. Yes. Recruited. So what exactly did they do? They recruited, right? Recruited participants, right? That's just the verbs. Okay. And what are the criteria for the recruitment? They were from Coed City softball leagues and their age range was 21 through 56. Okay. Okay. And ages. Okay. So now we have two sub events related to recruitment, that they were from intramural city football leagues and softball leagues and their ages. Okay. What is the next, the verb in this next sentence? After age range. Okay. Gave. And I gave consent. Yes. Gave consent. Okay. Okay. And the next one? Set up. Set up what? Table. Okay. Set up a table with. They advertised free drinks. Sports drinks. Okay. So there's two verbs in that one. Okay. Okay. Next verb. Competing. Okay. Players who had just finished competing. Okay. That's actually the, the head, it's a subordinate clause that's been moved up. So that's actually the noun phrase. Okay. Players who had just finished competing in one or two games, war is actually, we're offered is the verb. Sorry, that if you're not a linguist, you might not see that. But the actual verb of this sentence is we're offered. That also describes what the researchers did. Okay. What else did they do? First, the next one? They were shown. Yep. Oh, and I forgot the second part. Asked if they wanted to participate. Sorry. Okay. Okay. And then they were shown. Okay. We're shown posters with circles. Right. Okay. And what happened next? After they were shown the posters with circles? They were asked to select the circle. Okay. Okay. Next one. So the next sentence doesn't this has a descriptive sentence, right? It doesn't say what they did. Just describes the size of the shuffle. Okay. And they would report. They reported participants reported how many times they had been at bat. Okay. And actually other things. So how many times they've been at bat, the number of hits and walks. Okay. A number of hits. And all the other stuff that's missing from the other text. Okay. Which is fine right now. Okay. And anything else happened here? The next paragraph? It computed the next one. Mm-hmm. Computed what? Computed the batting average. Okay. Okay. And they say how that was computed. And what is the final verb? That information. Okay. They got demographic information. And one or lost. Okay. So these are essentially the decisions, the events that are described in this method section. Okay. Is this enough information to be able to describe or to actually reproduce the study? That's the open question. Well, actually, do you think it is? Can you reproduce the study with just the information that's been given here? Here as in the spreadsheet or here as in the text that was given in the paper? In the text that was given in the paper? I'd say no. I'd say you could approximate it. But I think there are some details missing, like, for example, the exact sizes of the circles, the kind of the spread of the softball leagues and the cities, towns, whatever it is that they went to to recruit. I think a lot of the details of the sort of practicalities of actually doing the research maybe. Missing? Okay. Exactly. So a good way to get at them, to get at the missing information is to take each of these events that we have and see what decisions were made so that those events could take place. So what are the decisions involved with recruiting participants? I'd say like the amount of participants that you want to recruit. At least your goal, like the actual number that you get is going to be out of your control. But how many you're trying to get to is a decision that you make. Okay. So you need to decide how many participants. Okay. And what else about the participants? Inclusion and exclusion criteria? Yes. Where are you going to get your participants? Yes. Where? And specifically even if these participants, you know, in Intermeerland City softball leagues, like those are two different things. So are you going to be going for players who are very experienced or are you going for players that are amateurs and how are you recruiting those? Yep. Okay. So that's a lot of information. See, when we start to break it down, then we see exactly how much information is not there. So why these players? Can we reduce it to that? Why intramural versus the other one is local, right? City football league? Why do I keep seeing football? City softball leagues? Okay. And city leagues? Okay. Over what time period? I'm sorry? Over what time period are they recruiting? Okay. It's recruitment. Okay. What else? This could go into the inclusion or exclusion, but the demographics, then especially here, we can see they recruited more men than female. Okay. Was the higher? No. Okay. Actually, I'm turning it into a question now. Okay. Okay. So higher number of men than women? So gender imbalance? Are we going to say it was a decision? We don't know. Okay. What else would be the recruitment of the participants? What other decisions were made? Do you already have a where? Okay. Whoever said where, what were you thinking? Like what city, what state, what parts of the city, all levels of where? Okay. Cool. And why does this matter? Well, if they were after softball people, they went to a soccer field, that would be a bad decision. And so this also kind of feeds into well, why did they go for the softball rather than one of the sports that they had listed earlier, like baseball, golf, tennis. Okay. So I guess there's no question here. There's no verb here, but that's another question. Why the sport? Okay. Cool. And that will become really, really relevant in a second. Okay. So COVID and tremoral ages, that all falls into recruitment. Okay. Gave consent. What decisions had to be made there? Debbie? Well, they should have had an REB. As soon as you talk to a person or a human being, you need research ethics board approval to be able to do this type of research, and therefore you need a signature from any of your participants saying that they're willing to actually participate. So there's like a lot of paperwork that goes with cave consent that I don't know. I don't know if they did or didn't. Okay. So we need pretty much the details of consent, right? But that's actually in the missing. So what are the decisions involved in giving consent here? I would say actually the means for providing the information because you can do like just verbal consent. Some ethics boards will let you do that, but you also need to decide how much information to give and when to give that information and in what kind of language as well. Yeah. Okay. Okay. So all of that's going to go in the missing. So hold on a second. So we need to, so they made decisions about how consent would be given. Right? Can we go with that for now? Okay. That works for me. Okay. Well, we'll really get into it in the next column. Okay. I'm sorry. Okay. Sounds good. A set of table. That feels like they needed a decision about how they were going to implement their consenting process. Okay. Is that what the table is about? Only the consent process? Let me start with. I think the table has to hold the drinks. Okay. So what's unclear here is exactly what the table was for, right? From the text. Right. That and where is the table going to be located? You know, its visibility, how large is it going to be? Is it going to be bright and flashy? This four. Okay. What the table's properties are. Okay. And why? That's the point with being flashier. Not right. Okay. I'm advertising free drinks. What were the decisions there? Are they using a sign? Like that says free drinks? Like what's on the sign? What motive advertising? Okay. Okay. So how to advertise? What about the most obvious one? Why use drinks as well to as bright? Well, there's sports drinks and they were playing softball. So maybe they think that that would be attractive to the people playing. Okay. But maybe there's also questions about like the brand and the size of the bottles and the flavors and so on. Yeah. Do you think that might have an effect on participants and the responses? The specifics of the types of drinks or just having drinks. I don't know. Yeah. I mean probably. Right. I don't know which direction. Okay. But that once again goes in the missing. Okay. So they were offered drinks. Okay. And what decisions are involved there? Yeah. Actually, there are more decisions that are offered in the that are part of the offered drinks part. That's the who does the offering, I guess. Okay. Yes. Who does the offering? Okay. And what else? What do you approach? I mean, you know, are the people are offering the drinks, picking people based on how tired they look, you know, people who look friendlier, those sorts of things. Yep. Yep. Also, are you going to offer them a cup with the stuff in it? Or are you going to give them a whole bottle? I was just going to say that with the times of COVID, I can't imagine going forward and not offering someone a full bottle that they have to open themselves as opposed to a cup. This paper is actually from, hold on a second. So my PhD supervisor was James Cutting. And the study was processed during the time that he was the editor. So that must have been the early 2000s. This is an old paper. The reason that I select these papers is this is short reports in psychological science. It's because most of these papers are one or two pages. So for pedagogical purposes, they are perfect. Because otherwise, we just don't have enough time to do the entire paper because there are many pages longer. So no, it's not the times of COVID, but yes. Okay. So how was offered? Okay. We're showing posters with circles. What are the decisions there? Size of the poster. Size of the poster. How many circles? Okay. That each of the circles is not an individual poster. I'm assuming that all the circles are on one piece of paper. Okay. How many circles on each sheet? Okay. What else? How the circles are arranged on the poster? Arrangement. I really cannot spell. It is really sad. Arrangement. Arrangement. What else? Color. Color. Yep. The color of the circle. Yeah. Yeah. Are they actual circles or are they dots filled in? Yes. Okay. Cool. That's a good one. I hadn't thought about that one. Circles or dots filled in. Yeah. Yeah. I mean, also that in combination with the color, right? Because if the color is black, it's one thing, but if it's rather a soft gray, which is what softballs look like afterwards, or baseballs look like afterwards, would that make a difference? We don't know. Okay. We're showing posters with circles. Okay. Ask to select the circle. What decisions are involved there? I guess are the circles labeled or does the participant point to, touch, give, list out the label? How are they saying their selection? Okay. How is selection communicated? Okay. How is the selection communicated? Okay. And I guess that's pretty much the only thing there. Participants reported times a bat number of hits per watts. What are the decisions there? How do they prompt the responses? Okay. Just prompted. But someone else, I forget who raised the question of, oh no, no, I guess it was for the one or lost, whether that matters or not. Okay. How responses are prompted? But then we also have the obvious one, right? Well, do they write them down on a piece of paper? Or do they just say it and someone has to record it? Oh, that's a different one that I was thinking about. But yeah, that's the one. There's no reason why you should read my mind. Okay. Are prompted how they were reported. How responses were reported. Okay. Also, why those metrics? Yes, that's the one that I was thinking about. Why those metrics? What's the justification? Yeah. Are these cumulative or just for the most recently played game? Right. Good one. Cumulative. Okay. How do you spell cumulative? C-U-M-M. C, okay. C-U-M-M-A-L-A-T-I-V, right? U-L. Thank you. In my defense, I juggle four languages on a daily basis. So I really, really can't spell. It's really pathetic. It's the cost of multilingualism. Okay. Right. So none of this is justified in the paper. It's just kind of given to us. The computed batting average, I guess that follows right from the previous one, right? And then they got the demographics information and then whether they won or lost. And that I think goes to recruitment and won or lost. Someone raised the point of why does this matter? So deciding on info on winning or losing. Yeah. Okay. So that's that step. Okay. Now let's move. I'm going to spread this out here. Okay. The next tool set and the final one in deciding what information is missing, though I think you have a pretty good idea at this point, are what we call the universal interrogatives. And I'm just going to stick them here on the side. I know it's big words, but these are actually words that you know very well. When, where, why, and how. Okay. So in determining what information is missing from these decisions, given the verbs that were given here, let's ask questions like who did what, when, where, why, and how. So when it comes to recruitment participant, the recruitment of participation, who, what are the who's in this event? The researchers and the participants. Okay. The researchers and the participants. Is that it? I would say also the community from which they're drawing participants. Okay. From which drawn. Okay. One of the things that shows up here, specifically at this point, is that we're assuming that it was only the researchers who participated in the actual study. Is that the case most of the time? Or do we have assistance? There could be assistance. And those could be any level of like undergrads, grad students, hired lab techs. I guess I would call them generally study staff. Okay. Right. We can call them study staff if you like. Okay. Also, since this was out in the community, it, it's possible that there are other people involved, like the, like if they're staff at the field, there may have been conversations about, you know, permission to set up and maybe some staff helped recruit. We don't know. Yeah. Yeah. So, one of the things that happens when you look at science as narrative, and you start asking these questions, specifically the universal interrogatives, is that a lot of the invisible labor in science becomes very, very visible. So you start to see what actually happened during these events, and you start to ask, who did this labor? Who did this work? And depending on what lab you're in, a lot of the time, the people who signed the paper and who got authorship are not the people who did the work. As yeah, that's one of the things that happened. So we have a possible staff at location, the study staffs. Okay. And what they did is they recruited the participants. Okay. When do we have a when? We don't have a year here, right? And this one, I think we only have it in the actual paper. Yeah. All right. The other who that we have, that's important information is these names. And yeah. So what do we know about these people from these names? Where they work? Yep. We, okay, that's the University of Virginia, but what do we know about them right off the bat, just by looking at their names? We're talking demographics here. It appears at least judging on names that they're probably, it's probably one woman and one man. A woman and a man. Yep. And that they're likely white. Right. They're likely white. So that's part of the who. Okay. Let's go back to, oh, here we are. Sorry. It did click. Okay. So that's the other who. In the context of this study, this is a study on baseball. Okay. And we found out that they were at Virginia, Virginia Tech or something, right? Okay. With these results, the way that they're, or actually not the results, but the way the study is done, I think there are assumptions made that this would apply to people everywhere. Right? Yes or no? Oh yeah. Exactly. This is just a really quick point about why who matters so much. Because when we actually look at who the researchers are and where they are, we realize they're making assumptions about the entire human population based on who they are as individuals. And as we all know, that's pretty dangerous. And it's one of the things that fortunately open science has been paying attention to. Okay. So we have the when. This is actually the 90s, I believe it was a 2000s or something. Okay. Where? Okay. So they mentioned Charlottesville, Virginia, which now makes sense because the authors are at the University of Virginia, which is in Charlottesville. Which is in which part of the US? East Coast. Yeah. East. Yeah. East Coast US. Okay. Actually, I was going to put that in here in the, I was not going to write that there. Put it in the wrong column. Never mind. Okay. And that doesn't work there. I have a Windows keyboard here for some reason. And let's do that. And the keys don't play nicely with my Mac. I'm do typing. No, it's going to be in here. There we go. Okay. Copy and put that for here. That's our where. We have the when. That was early 2000s. We have the who. So about what was done in terms of the recruitment, we have all the decisions, right? Pretty much inclusion, exclusion. Okay. Consent. Yeah. Okay. That's over there. Yes. Consent is the next one, I think. We forgot how. Why and how? Okay. Why? There are many, many, many why's that aren't answered here. So we have why these players, why intramural? Okay. And I guess we have, we've listed them. And then we have the how. What do we know about the how? How were these participants recruited? Do we know? We don't know. Okay. Good. The next ones. So we can pretty much go through this process of asking what, who, when, why, how, where to figure out what information is missing. So what we have here is, do we have anything else missing here for participants, for the recruitment of participants? We just don't know how they were recruited. Monica, may I ask a logistical question? Yes. With these interrogatives, something that I've been a little struggling with here is where to draw the line between these verb phrases and decisions about in at what level to ask these questions? Because it feels like sometimes I ask the question, but then it's like, it's in like five different verbs sets. Is that normal? Like we want to work across the verbs or work one verb phrase at a time? It's unavoidable to work across. So the reason that we extract the verbs is because it gives us a place to start. Okay. That's pretty much what they are. But of course, they're all interconnected because these events are interconnected and they overlap. But the important question is at this point, just at this point, are you getting a better sense of what these researchers did and what information is missing? Awesome. Thank you. Okay. So I mean, but that's the question. Are you, if you are, then it's doing its job. Yes, I am getting a better sense of, I was just trying to figure out like, and I need to be like restricting myself to one row at a time. Or no, as you can see, even I've done this a bunch of times and it gets a little bit hard because the events are so interconnected because they're narratives and in narratives, it all occurs on several different layers at the same time. Okay. So we need more information on how they were recruited. We don't have that information. And when it comes to COVID intramural cities, softball, league stuff, that's a big why question, right? And I think we already have that one. Consent. Yes, we are at consent. What is the information that's missing with consent? And I think someone earlier gave a pretty good description and I said, we'll go back to that. So do you want to revisit for me please? Was that might have been me? Maybe. Was it when I said like, how much information you're giving and like what kind of language you're using and whether you're getting verbal consent or having them sign something? Yes. So here they give nothing. So we need all information on consent. And what is that information entail? So it's what language? Okay. Was this oral or written? Was there an IRB process, right? Not an OR process. Did anyone here watch Deep Space Nine? The ORB experiences? Star Space Nine. Yeah. Okay. Was there an IRB? Okay. Going back to invisible. I'm sorry? What was the script? The recruitment script? Yes. Okay. Script. Yes. Where are the who's in the consent? In the consent process? Besides the participants? The IRB? Yeah. Okay. IRB people. That's invisible labor. What else? The PI. Yeah. And then. Think about this in terms of who actually did the work. And I was just going to say that the actual people getting the consent, carrying out the script and. Yes. Yes. So let's call it the staff labor. Yes. Okay. And they are particularly important for recruitment since this is happening in the field because their personalities matter. Right? If you're recruiting in the wild, you need people who are comfortable talking to strangers. Otherwise you're not getting anywhere. Okay. So staff labor. Okay. That's the information we're missing. Okay. And the setup of the table. Is it just the setup of the table that we're missing? Or what else are we missing? I guess I just think the logistics. Where did the table come from? Okay. Do you have a tablecloth on it? Do you know? Yes. Did you have a card big enough to get the table there? Or did you borrow it from the people who are already in that? And I guess some of that stuff does matter because we also kind of put in where the location of the table was in terms of the setup. Yeah. Yeah. The logistics of the setup because they matter. Yeah. There's, for another talk that I gave a few days ago, I think there's a great quote and I'm trying to see if I can pull it up really, really fast because it is so important. Actually, it's in the abstract. It's in the abstract that studies will not replicate or will not reproduce because researchers do not think to document these tiny details because they think they're inconsequential, but in the end, they end up being the reason why you get an effect or not. So you don't know. That's why we document. Okay. So we need all of the logistics of the setup. Yeah. So that would involve things like tablecloth, transportation of equipment. Yeah, all of it. Okay. Who actually did the setting up of the table? Yes. Come back to that notion of invisible labor. Yeah. Who did the setup? Yeah. There was something about getting permission to set the table up and arranging all of that. Getting needed permits. Yes. Who made the phone calls at seven in the morning to call Parks and Recreation to actually get permission to do this? I don't know why you assume that if this is two white people that they just didn't walk onto the field themselves and didn't ask for permission to do anything. Or did they get permits? Yes. Maybe the reason that they only have that many participants is because they didn't get permits and then Parks and Recreation staff showed up and said, hey, you're not supposed to do this. Get out. We don't know. It's not in the paper. Okay. Cool. But are you starting to see that once you get into this mindset of asking who, what, what is the information that is missing, that it just kind of snowballs in your head? And the reason that it does that is because you're using exactly the same mechanism that you use when you're reading a novel. So what's a good novel versus what's a bad novel? A good novel is where the things that the researchers, that the characters did make sense according to what we know about them or according to their goals. When the events described in a novel do not make sense according to these constraints, we usually call that a very holy novel, not holy as in the Pope or holy as in Swiss cheese. And we call it a bad novel in general. So when we use exactly this skill set, this tool set to look at documentation of the paper, we find the holes very, very quickly. And it's snowballs in our head. It's kind of like once you see it, you can't unsee it. So in many ways, I'm not really teaching you, I'm making you aware of something that your mind already does automatically. And as a psychologist, I really wish that in open science, we use so much more of what we already know about human cognition to make this easier on us and to make it work better. Okay, free drinks. What information are we missing? What's missing? What kind of drink? What kind of drink? Okay. And what about what else besides the kind of drink? The variety of different kinds. Okay. Was there a variety? What about the size? Yeah, that's missing. Yeah. Okay. And are they allowed to have more than one? Because it's, I don't know if that's completely clear either. Yeah, more than one. Okay. Did we already have wide this drink? Or what other drinks were considered? Okay. That kind of stuff. Okay. And then we're offer drinks. Is there any information missing there? What if someone wanted to participate but didn't want to drink? Okay. Right. Forget my misspelling too. Without a drink. Okay. Sure. Okay. Asked if they wanted to participate. What do we need there? What information is missing? Who did the asking? Okay. Who did the asking? Okay. What else? Did they have a set script for how they asked people? Okay. What was the script? Which then means of course, how was it developed? And by whom? But after a while, you start asking pretty much the same sets of questions about every single step. So recruitment. Who did it? How was it designed? What were the criteria? Blah, blah, blah, blah. Okay. How was it developed by whom? Okay. Yeah. Oh, right. The other one is how was it decided who would be doing the asking? We need to hurry up because I need to send you guys off to do your own. I think you get the general idea, right? Yes. I guess I just had a question about what the end result is, is that we're looking for how to create something that is reproducible by asking and ensuring that we have answered all of the possible decision-making variables, missing ideas that we would do in our own experiments. And for something like this, I guess I just see it as why wouldn't you just videotape it? You wouldn't necessarily have to verbally describe it, but you would have a video recording of the whole interaction. I could not agree more. In fact, so this methodology comes from my work on teaching people how to read science. And I realized very, very quickly that I could describe what I did in the classroom, which is kind of bogus because a classroom involves all of these interactions that are completely unpredictable or I could just record them. And at this point, I have dozens of video recordings of exactly what happened in the classroom of my sessions. And I do not understand why they're not recorded. I could not agree with you more because then people could see. I completely agree. Though even that's just the actual instantiation of the experiment. What about all of the other stuff? All of the other decisions. So yeah, it's a partial solution. It's not all of it, but it would definitely go a long way. Okay, so let's finish this really fast because then we have one final question that we need to address. Okay, ask to select the circle. Right. So how is selection communicated? That actually goes over here. The rest is just criteria, right? Why these metrics? Okay. And whether other metrics were considered and why were they discarded? Why discarded? Okay. Cool. So now that we read this again, do we have a better idea of how to do this experiment? Or at least what information we need to obtain from doing this? Yes? No? I hope so. Yes. Yeah. Okay. Okay, sounds good. So how many people do we have here? Actually, Chris, so you probably know how many more people we have, right? We've got 12, including you and me. Okay, fantastic. I am going to go to another thing here. We have another page. Copy link. I'm going to put it in the chat. And I guess pick the paper pane because I think that one's pretty cool. I don't know. Here we are. Chat. Here we go. Here is the, and it didn't copy it. Share. Yes. Copy link. Thank you. Please. There we go. Okay. Go to the spreadsheet. Crystal, can you put everyone in pairs then? You said there were 12 people, including me. Six, seven, eight. Not including you and me. Okay. I guess we just lost a few people. Okay. So I guess, yeah, groups of two and three. Okay. Yeah, put them in breakout rooms. Yeah, and breakout rooms. And just according to the number of your breakout room, pick a group, read the paper, and just do this exercise together for the remaining half hour. Okay. So one thing to note, folks, is make sure that you've opened the spreadsheet because sometimes the chat disappears when you go into breakout rooms. Okay. Okay. How was it doing it on your own? It was an interesting experience. Yeah. Interesting, useful or interesting, pointless. Useful. Useful. Okay. If any parts of it feel pointless, please let me know, as I was just discussing with Crystal, I've only done this face to face. So it's a really, really different setting as everyone who's taught knows before, exactly. And you rely so much on your students' expressions to see whether you need to rephrase things or reframe or redirect. And here I have pictures and names. So if I can't read your face. Okay. So let's see. How far did we get? You got as far as verbs. Okay, I can see that. Okay. So one word of advice, group one, I don't know who you are. So don't take this personally. You should probably write out a little bit more context when you list your verbs. Not just the verbs. Okay. So group two, right, that's a little bit more context. Yeah, pain condition. Right. And return. Okay. Great. So we really do need lots more time for this part of the exercise, I think. I was hoping that we'd have time to compare the missing information, but I guess we didn't have time to do that. 20 minutes wasn't enough. Okay. Any thoughts on anything that we discussed so far? Maybe realize that writing when I write papers in particular and others, it really should be much longer and more descriptive than it is. Yeah, it really, really should be. One of the things that we're advocating for is for methods sections to become methods document, its own document, and that it should be done in excruciating detail. That's, yeah. So this is part of a two-part exercise. The second part involves you actually describing your own research in excruciating detail. And then seeing, of course, if somebody else can replicate what you did, we start off with really simple things. But since we really, really, really didn't have enough to do that, let me show you a really quick video. Okay. That is right here. And if you've watched it already, just enjoy it one more time. It's pretty much the point of what we're doing right now. As an FYI, Monica, right now, we only see this sheets. Yeah, I know. Okay. Just double check. I know. It's on my other screen. I'm copying and pasting right now. It gotta be complex, then. We can ignore that part. I'm in Germany, so I can't deserve my ass. You're not even making any sense. Sorry, you ruined it on purpose. You know what? I'm hungry. I can really go for a peanut butter and jelly sandwich. You guys think you can write down some instructions and teach me how to make peanut butter and jelly sandwich? Yeah. Step one, get two pieces of bread out. Get a butter knife and get some PB. Take one piece of bread, spread it around with the butter knife. No, dad, with the peanut butter. I'm just doing what it says. It says, take one piece of bread, spread it around with the butter knife. Hold on. Get some jelly, rub it on the other half of the bread. No, dad, open the jelly. It doesn't say to do that. Put the breads together on top of each other. All right, take a big bite. Sorry, I had to make it extremely specific. Oh, good. I'm starving. Take two pieces of white bread out of the bag. Take the lid off the jar of peanut butter. Get a butter knife and stick it inside of the peanut butter. With the knife, scoop a bit of peanut butter out. That means a lot. A bit means a lot? In my world. There we go. Doing better than before. Open the jelly jar. Squeeze it onto the other piece of bread. No. Done. Get two pieces of bread. Get some peanut butter. Take the peanut butter knife. Open the peanut butter. Put the knife in the PB. Get some jelly. Open the jelly. Squirt the jelly onto the bread. Take the butter knife with the peanut butter on it. Wipe it all over the piece of bread that's blank. Take the butter knife, rub the jelly all over the piece of bread. Put the two pieces on top of each other. Take two pieces of white bread out of the bag. Take the lid off the jar of peanut butter. Get a butter knife and stick it inside of the peanut butter jar. With the knife, scoop some of the peanut butter out of the inside of the jar. Spread your scoop of peanut butter onto one of your pieces of bread with the knife. No. Squeeze some jelly onto the other piece of bread. Spread the jelly on the bread with the butter knife. Put your two pieces of bread, peanut butter and jelly sides together. Done. Okay. I think you get the idea, right? This is so much of what happens with our research and that's the problem that we're trying to solve. So this is one of our first attempts to really, really get into the nitty gritty. It's hard work. It's very detailed. It's tiring. And I wish there was a way around it. But if all we have is this paper, then our only outlet, our only way to do this is to literally go line by line and see what they did and then try to figure out what is not there. And hopefully the researchers are still alive or in academia or findable so that we can get any missing information because otherwise what we end up with is pretty much the instructions for how to make a peanut butter and jelly sandwich as we see here. So that's the general message of this session that we want to avoid those peanut butter and jelly sandwich instructions. Okay. We are about out of time. Does anyone have any final questions? Any final thoughts? If anybody wants to reach me for anything about this method, feel free to reach me at this email address, please. Here you are. And otherwise, thank you so much for coming, especially for sticking it out to the end. I know some people left because it's hard work. It's grunt work and there's no way around it. But so yeah, thanks for sticking it out. And there's my email address. Okay.