 Up next, Misha Tplitsky with Debiasing Peer Review with Voluntary Anonymization. All right, so happy to present this joint work. This is with Ina, who's here in the crowd, and Daniel, who couldn't be here. And actually, I need a little clicker. All right, this works, great. So the topic is, I think, dear to many of our hearts, it's peer review and in particular bias. So there is a long literature, many of you guys have probably seen maybe all these studies looking at bias in peer review, measuring it through anonymization, or blinding, we'll call it anonymization. And then this literature, people have looked at a number of aspects of identity, race, gender, et cetera. Arguably the most robust finding is around prestige. So that'll be our focus here. And so we have this literature, it seems to be fairly robust, anonymization seems to reduce prestige bias, but it's very much not universally adopted, right? Especially to those of us outside the social sciences, or those of us in social sciences that may come as a surprise, but in practice, especially in physical sciences, this is somewhat rare. Most journals are single blind or single anonymous. So why is that? Well, a bunch of reasons, right? So maybe it's kind of just organizational inertia, maybe some authors don't want this, probably some authors don't want this. Maybe there's some benefits that we get from single anonymization mode, like maybe spotting conflicts of interest. And people also say, well, who cares anyway, right? You can always Google and people probably do, so it's not really much of a, we shouldn't worry about this. So that's sort of one kind of set of reasons, it's a conventional one. In addition, I think there's a compliance issue. So when we talk about anonymization, it's like someone should probably check if it was done correctly, that someone maybe at some point in the future will be an algorithm, but at least for now it's not. And at least one company that will be showing data from in a second found that it's actually extremely costly to do this checking, right? They estimated, the Institute of Physics Publishing estimated at almost 2,000 of editor hours per year, I think, that it cost them, right? Okay, so it's a little tricky, maybe very tricky to implement, so why not do the optional kind of voluntary version, right? That seems attractive, thinking emoji. Let's see, so it seems attractive, but, right? And there are a bunch of kind of theoretical reasons that we might expect this not to work. So I think main, probably main reason comes to mind is strategic behavior, strategic responses. So probably people benefiting most from the system are probably gonna be the ones who wanna showcase their identities loud and loudly. And so if you do anonymize, it kind of shows you're not one of those people, right? So anonymization is a signal for kind of your identity. And we can even kind of roll that game forward where it's like, okay, the second, most prestigious person doesn't anonymize, all right, then the second most prestigious person thinks, well, I'm the second, so I also benefit from not anonymizing. And you kind of play that game forward and then equilibrium actually no one should anonymize, right? That would be like the economic prediction. But of course there are some limits to strategic behavior. In particular, in this kind of setting, it might not be so obvious whether your identity kind of helps you or harms you if you are a super self-interested person, which certainly I'm not, but I'm told some people care about their publishing outcomes. So whether you benefit or not from the system may be a little kind of noisy, hard to predict. And also how exactly kind of frame the choice probably matters, right? Like is this a requirement? Is it like strongly encouraged? Is it just like an option? So maybe like culture and other kind of factors come into play. And also you might worry about unintended consequences, right? If the name of the author is hidden and it's like a noble lawyer, maybe I want to review that paper immediately because let's say I want to learn from it or whatever. But if I don't see the name, maybe I'm less enthusiastic, right? So journals might worry about these kinds of, does it affect the recruitment of reviewers and authors? So that's a theory. Fortunately, we do have a little bit of data, right? Already. And so nature, journals tried this option I forget exactly when this was rolled out. This is kind of summarized in this paper. They rolled this out across their journal portfolio. I analyzed about 110,000 submissions. And the paper is a bit, it's worth reading. It's a bit kind of, I guess, ambivalent is what it sounds like. It's sort of, there was not a 12% take up of this option. You could say it's low. It gets debatable. They didn't find an association with gender. They defined kind of some differences by country. A little bit of difference by prestige. But the kind of crucial sort of takeaway from this project was that the design wasn't really designed, it wasn't designed to enable causal statements. So kind of nature rolled us out all at once. And it was hard to kind of conclude, does this causally reduce bias in any way? So that's where we hopefully come in and contribute to. So we're working with the Institute of Physics publishing, which around 2018, started this move to double blind, double anonymous review. They ended up making it an option. And so the option is framed as you see here. Sort of, as an author, you can do that. We're not gonna check. And up to you guys, right? So here's a bit on the data measures. So we're gonna be studying data from 57 journals in IOP's portfolio, covering, as you can tell, a huge number of submissions from 2018 to 2022. The journals were, like the policy rollout was staggered. So it was kind of set of journals a month that this option was enabled on, which is kind of perfect for kind of making causal statements, right? So that's really the hearts and the eyes light up. And when we're gonna be talking about the effect, the causal effect of this policy, not the effect of anonymization, right? What happens when we enable authors to do this? We're gonna measure author prestige in the following way, with citations, you got up to submission year. We can kind of quibble about what the right kind of bins are. Many, I believe a third of the authors in our sample have zero citations of time submission, third of first authors. So we played around with this. It's gonna turn out to not matter all that much. We use self, mostly self-reported gender and country, and we'll be looking at review positivity in a kind of binary way. One, if it's you got accept or R and R, and zero, reject. All right, so, maybe I should be pointing here. All right, so who chooses to anonymize? So, the red line will be our reference category, which is a person of quote-unquote low prestige, so low citations, who is based in the USA and is male. So relative to this kind of hypothetical person, we see that people who are middle prestige anonymize a little less, people who are high prestige anonymize even less. It may be predictable to some extent, but note that it's not zero, right? It's not like people are not behaving completely strategically, one could argue. We also see that patterns by country are pretty prominent and actually different from nature's experience. So, we see that China and India where a huge number of submissions to these journals come from actually anonymize a lot less than authors from countries like Germany and US, where if you go off the literature, you would probably guess they'd benefit from their identities being visible. Okay, so from this, we kind of conclude that there's some strategic behavior going on, but it's not really what you would necessarily predict, and so it's probably not completely strategic. Okay, what are the results of this policy? Okay, so we're gonna look at how positive are reviewers, peer reviewers. The panel on the left will show actual rates before and after the policy, before the policy will be solid line, after the policy will be dotted, and then maybe the even easier panel to focus on is the change, like what's the actual change in positivity? And we see that for low status authors, those are very few citations, reviewers got a little bit more positive, about 2%. For middle status authors, it's kind of like a kind of boringly significant, it's point estimate is also not huge, and then for a high status authors, kind of even less of an effect. These are also the authors that don't anonymize this much, so the behavior changed less for them, and you kind of see less of an effect. But it's interesting that even with this anonymous policy, and this we can kind of say kind of causally, it increased positivity of low status authors. So then we're gonna look at acceptance, arguably even more important, and here what's interesting is effects actually get even a little bit bigger. So looking at low status authors, change before and after, we see that low status authors are now accepted about 5.6% more often, which is kind of not trivial I think for like, many of us will love that, right? Middle status authors are accepted a little bit less, and high status authors don't have this much of an effect. But again, to emphasize this is a causal effect of giving people the option to anonymize. So we might also worry about unintended consequences, or I did this effect to review recruitment, journals, they're certainly gonna care about us a lot, especially if you're like maybe not at the very top of the pyramid or whatever. And probably we found no effects. So reviewers accepted invitations at about the same rates. We didn't find a change in the prestige of the reviewers, again, measured with citations, and similarly for the authors. So that's kind of those sort of good news. So to sum up, all right, so the conclusion is I would say fairly straightforward. So volunteering organization, this seemingly attractive, cheap option to reduce bias, by which we maybe guess wouldn't have worked, actually seems to work. Why that is, is probably because, whoops, it's probably because it's not, how do I go back? It's probably because the behavior wasn't quite strategic, like maybe it would have feared, so whether you anonymize or not is not really all that reliable a signal of who you are. So review positivity went up to 2.4%, acceptances went up 5.6%, medium and high-sided authors were affected less. And we found no evidence of unintended consequences, great. So there are a bunch of open questions here, like what exactly is the mechanism, like is it actually the standard kind of cognitive bias or if I see a famous person, I assume it's better work or whatever, or is it maybe that reviewers know that this policy is in the field and so they kind of punish people who don't anonymize, because they feel like they should. So there could be, there's someone being guilty about exactly how this is being caused. And I think another question is the long-term effect. Like this is, we're speaking relatively short-term here, would this policy hold up that people kind of reach some new equilibrium? Would it be as good looking as this one? It's kind of unclear. But the sort of, I think the biggest takeaway is that this bias is very common. Here's essentially a free way to reduce it. And so we should probably experiment with that a lot more and hopefully replicate this in other organizations. Thanks. Dr. Torelli, Open Science is a really great work. I'm curious if you could share more about the policy itself, how it was implemented. As reviewers, we all know that the one untold secret is that double anonymization works as long as reviewers don't de-anonymize the author by looking them up and trying to break the double-blind clause. So was there anything put into place that if anything, if this still occurs, you may suggest that the nominal effect could be even larger if these practices didn't exist. So I'm curious about the policy implementation and anything you have to share about that. Yeah, going off of what you said earlier in the session earlier, it's also like I think I kind of say it's like a beer and like two hours of going into the details. A lot going on here that I'm kind of not going into. So one, so first, IOP tried to roll this out in an enforced way, like kind of usual quote unquote. And first found that it was too expensive, right? So first there's like five journals this was tried in that are not in our data set because of this kind of more complicated history with the policy, right? So there's like that aspect. In term, then the wording, there's like, I think the messaging is really important. How exactly was worded? I showed kind of one snippet from the litter from their kind of website. There was like probably a few places that was messaged and maybe was slightly different undertones or overtones, whatever. There was like a template given to authors or like a checklist, like make sure you check all these things before you submit. Oh, by the way, it's optional. There was kind of like a little bit of a like nudging and like strong suggestion vibe to it, I would say. Then there's like kind of how it was like which journals were chosen to go first like how this the staggering worked also has some quirks to it. Yeah, you know, happy to chat about all that more. I kind of, we sort of suspect it doesn't affect our results is kind of our, you know, like we excluded the thing that seemed like what kind of would be quite different from what we actually studied like those five journals, but there are some quirks going on here. No guidance, explicit guidance for reviewers. Oh, in terms of preventing them from like a de-analyzing. Yes, yeah, I believe there wasn't at the same time the policy was announced publicly. So I believe the reviewers weren't like sent an additional instruction like, oh, by the way, please don't Google this person. Yeah, everything like this, I think everything was designed to be like minimal. Like they found that it was very costly. They're like, hey, we're just gonna give people the option and then hands off. Like we're not gonna change anything else. That was my impression. Yep. Thanks, Richard. Yep.